Sorry, the language specified is not available for this page

    How To Build A Serverless Alert Email Notification System Using Machine Learning

    April 9, 2018

    As an AWS partner, we receive a lot of email messages from AWS including many that need prompt attention. But with so many emails, it is very hard to detect which emails need immediate attention and which ones can be handled at a later time. To deal with the volume, I’d like to present a serverless system that sends an alert to a Slack channel whenever an urgent AWS email arrives.

    Before we get started, here is the overall architecture of the system:

    First, I trained an LDA (Latent Dirichlet Allocation) model using all previous emails and stored it in a S3 bucket. In the process of training, the model groups all previous mails into as many topic groups as specified, and you can review the generated topic groups to find out which group(s) should be considered to be alerted.

    Now, a Lambda function regularly checks to see if there are any new email messages in the mailbox. If so, it sends the content of the new email to another Lambda function that feeds the content into the trained model. The model generates a proportion value for each group that shows how likely the email belongs to each group. Based on that information, the Lambda function sends an alert to Slack if the returned proportion values of groups to be alerted are high enough.

    This system also provides a RESTful interface using API Gateway that invokes the same Lambda function to run the trained model against the given email content and sends an alert to Slack if the given email content is considered to be urgent.

    Next, let’s take a look at the Jupyter notebook to go over how to train the model with scikit-learn library.

    1. Let’s create a DataFrame using all previous emails. 
    2. Transform all email contents in the above DataFrame to a “document-term matrix” that represents how frequently each word is used in each document.
    3. Create a LDA model and train it with the above “document-term matrix”. Here, we choose 10 for ‘num_topics’ and set a fixed value for “random_state” seed to guarantee same results from repeated training.
    4. We can check topic groups generated by the model to see what words are mostly used in each group.
    5. Process the trained model against all emails to generate a proportion value for each group that shows how likely the email belongs to each group.
    6. Store the trained model in a S3 bucket for future use against new emails.
    7. Create another DataFrame using all emails with their generated group proportion values.
    8. Now merge two DataFrames, #1 & #7. You can see each email has topic groups (t00~t09) with their proportion values (p00~p09).
    9. This is a code snippet of how to use the trained model to get the topic group proportions against new email.
    10. The function “predict” receives input values of the trained model object and a new email message to be processed, and returns a list of topic groups along with their proportion values like below. It shows that the given email message most likely belongs to topic group 4.
    11. Finally, you can send the email as an alert to Slack if the topic group considered to be alerted has the highest proportion value. Here, “Possibility” is the proportion value of the topic group.

    In summary, this is how to set up a serverless notification system that sends alerts to a designated Slack channel whenever an urgent email arrives in your mailbox. I used scikit-learn LDA model to train a model and picked up (a) specific topic group(s) for alert notification after reviewing the classification result. I stored the trained model in S3 bucket to use for new emails in a Lambda function, which supports serverless architecture.

    Other Posts You Might Be Interested In

    Cloud Governance at Different Scales – Part 2

    Moving up in Scale In the first part of this series (read it here), we looked at cloud governance when an organization is a small number of accounts.  As an organization... Learn More

    AWS Lambda: Taking the development world by storm

    Lambda, AWS’ server-less compute service, launched at AWS re:Invent 2014, is popping up everywhere and revolutionizing how applications are architected. Major League Baseball Learn More

    Sungard AS Helps Support COVID-19 Research via Folding@home

    No one really knows the exact origin of the phrase “Necessity is the mother of invention,” but it is certainly not a new concept. Regardless of its origin, the concept is... Learn More