Logo
RSS Feed

👈🏼 Back to
System Logs

AWS Logs


Cloudtrail , AWS CLI

Created: 28.09.2020

This document is devoted to AWS logs. Several examples are analysed and useful fields are marked.

Audit

Login activity

For admin account go to IAM -> Credential report to see all the users and the following information:

  1. user name
  2. arn (acc N + username)
  3. user_creation_time
  4. password_enabled
  5. password_last_used
  6. password_last_changed
  7. password_next_rotation
  8. mfa_active
  9. access_key_1_active
  10. access_key_1_last_rotated
  11. access_key_1_last_used_date
  12. access_key_1_last_used_region
  13. access_key_1_last_used_service
  14. access_key_2_active
  15. access_key_2_last_rotated
  16. access_key_2_last_used_date
  17. access_key_2_last_used_region
  18. access_key_2_last_used_service
  19. cert_1_active
  20. cert_1_last_rotated
  21. cert_2_active
  22. cert_2_last_rotated

Service usage activity

Also, you can go to IAM -> username -> Access Advisor tab to see the services that this user has access to and when these were last used.

CloudTrail

This is an example log from a AWS CTF challenge. I am going to use this one in order to get an idea of what is being logged and how this can be useful for investigations.

{
  "Records": [
    {
      "eventVersion": "1.05", 
      "userIdentity": {
        "type": "AWSService",
        "invokedBy": "ecs-tasks.amazonaws.com"
      },
      "eventTime": "2018-11-28T22:31:59Z",
      "eventSource": "sts.amazonaws.com",
      "eventName": "AssumeRole",
      "awsRegion": "us-east-1",
      "sourceIPAddress": "ecs-tasks.amazonaws.com",
      "userAgent": "ecs-tasks.amazonaws.com",
      "requestParameters": {
        "roleSessionName": "d190d14a-2404-45d6-9113-4eda22d7f2c7",
        "roleArn": "arn:aws:iam::653711331788:role/level3"
      },
      "responseElements": {
        "credentials": {
          "sessionToken": "FQoGZXIvYXdzEFAaDEbnJXLefTT+kjlmKSKSBNgEUj8tJVL+szjaH5q2npYc2FIPgrLmfkRjK9KqtSW7+lo4WxteBTd77aeAcmIip4GceNBbU86zxGgS1IdNBzEOLnDw6biAzijG0Du/Qazx136qjy+kahHxPlR36C4y/0QrCUZpTFmP3uELsRIKkvhGvuBr6S10pTOZ+GjtUXN3iFV8Ea0KOo/fSP0d4LbZGwI957aJxs2I7N8ji/lKTfwPdq+sxXvSWnaOseinUxZUDS0zdI69CKb6C+qwhR5YTifqyuOvC9OoSlfcBN2FyHpRZf5Bd+Z+mPYTldbAvD/HcdbQo7U4jqlR2WGuXoBfwvypt/Kb6HtPp4g9O0HlTCc7Sb4uiJY81WMbaNFmmYXyj62gi+BN5QaA90YhWn9cU1x9gqt0uEgvSk/RdrwtulTtNyJcuuFvhlD1gaJHGc4eCoWApr+J9nrbPTvSo00sc8IYIVvwOi3NRsmP+ZA9aQOV/qg2L1cYxScQrQ/pKVOnYWJ4XuB0WL8gRYdo1bGI6LWGAtOV+fzVoXU0SfWH7UmPcvttqkWsv1Rr50pspmJneXeY6Ge1szNsvyHqmPFJj7GAXnEKl9xthHK3IaDc6HEprFqYw8SyQqYWGdpeism2S+V4XpIMbQJlC06BxyOg94H0Fiffzs8wwnCUpBU0s69X2tN8sxb4U3FqKl28mwCOuuaweZeGWq5MWxU7Fop2dmjaKN+u/N8F",
          "accessKeyId": "ASIAZQNB3KHGNXWXBSJS",
          "expiration": "Nov 29, 2018 4:31:59 AM"
        }
      },
      "requestID": "6b7d6c60-f35d-11e8-becc-39e7d43d4afe",
      "eventID": "6177ca7e-860e-482c-bde9-50c735af58d6",
      "resources": [
        {
          "ARN": "arn:aws:iam::653711331788:role/level3",
          "accountId": "653711331788",
          "type": "AWS::IAM::Role"
        }
      ],
      "eventType": "AwsApiCall",
      "recipientAccountId": "653711331788",
      "sharedEventID": "1d18bf74-8392-4496-9dc4-a45cb799b8b4"
    },
    {
      "eventVersion": "1.05",
      "userIdentity": {
        "type": "AWSService",
        "invokedBy": "ecs-tasks.amazonaws.com"
      },
      "eventTime": "2018-11-28T22:31:59Z",
      "eventSource": "sts.amazonaws.com",
      "eventName": "AssumeRole",
      "awsRegion": "us-east-1",
      "sourceIPAddress": "ecs-tasks.amazonaws.com",
      "userAgent": "ecs-tasks.amazonaws.com",
      "requestParameters": {
        "roleSessionName": "d190d14a-2404-45d6-9113-4eda22d7f2c7",
        "roleArn": "arn:aws:iam::653711331788:role/ecsTaskExecutionRole"
      },
      "responseElements": {
        "credentials": {
          "sessionToken": "FQoGZXIvYXdzEFAaDAhihig4wSQxSSiMjSKSBOW1B0tXzW8SIP7MtRZEikuZamVqb7hEQtmgX5LDeDzpOxpP4G9U8r3cZK74HNErY8W+Scri3Y8NVr/VO7MyzIXnpXiEDo10JfNYnA+urhxB+rYtPF6o8uzm40w/lqdq1/DyOkYYuFRtqWfS4Jy1t83KYAFTuX5IkTGekMidOkIcdnHjsZKQSqKs4U3MQ4doUG4nCyEERDv9yckTPq/R4fKEPTF1BN0jycyiSiR3OTPAWaLGMGxHthAKSA5IXosrPBAq0yD2HhSKZc0kbskCZyGOQCcmVAQK4IdEjyk4Lytc+PEDasvWiGoCQPEqlwwqFJm7EPm838MjCTi4ojN9nVYRP9hFYkXdvnVG6ScwoBfbg135vN1bqYgEKDncW780xUBRYwElM4Q2/6zv7DMj0UegbRJmAwtys4phLrItQqNLWPmBbW/pNgMYoT1IKfzDmuc27AyTHtL8t25hkYOLWZG19EYKm+XeHU9gbs0aDTPssjBFPZp7ssR35IHhgcw5m+etXSoXMxMuHNbVR46ZJ6uieR8roEZ8QfiIijyB8J7i2sH2JOY0HIOonbVYmqdtv++0D+1idTO+6ZbXJIKhEmDmZZDeenF2ZXQGH4PgA4udIdBXnhVLzVFkEKc1MWG3aAhuOhZLvnXkOpkn1XjwfDx3N0UyenOUR4x/gkY9+MEXMZAyO8Va2Y1vNZvnSCvTJtHDKN+u/N8F",
          "accessKeyId": "ASIAZQNB3KHGMG7YRUEW",
          "expiration": "Nov 29, 2018 4:31:59 AM"
        }
      },
      "requestID": "6b80a0b1-f35d-11e8-becc-39e7d43d4afe",
      "eventID": "457af3a9-0b1b-44ca-91e1-8f4a0f873149",
      "resources": [
        {
          "ARN": "arn:aws:iam::653711331788:role/ecsTaskExecutionRole",
          "accountId": "653711331788",
          "type": "AWS::IAM::Role"
        }
      ],
      "eventType": "AwsApiCall",
      "recipientAccountId": "653711331788",
      "sharedEventID": "5397e1a9-82c7-4a00-9b1c-e44cbd688aa1"
    }
  ]
}

To access logs for this region? you need to login and go to CloudTrail. If SSM is enabled, captures this activity as well. Similarly, session activity audit logs can be captured in S3 or CloudWatch logs.

⚠️ Not any CloudTrail log entry will have a resources part. For example, LookupEvents doesn’t.

💡 By the way, using CLI to lookup the events gets logged by CloudTrail.

CloudTrail events that are relevant from security perspective - https://www.chrisfarris.com/post/aws-ir/.

📘 BTFM

Here is my bash to retrieve CT logs for a all keys in the account, filtered by API call Decrypt. I also have a Python version and I am working on the Jupyter book for this.

loggroupname="aws-controltower/CloudTrailLogs"
for keyname in $(aws kms list-aliases --profile ProfileName | jq -r '.Aliases[].AliasName')
do
    echo "keyname is \e[92m$keyname\e[0m"
    key_arn=$(aws kms describe-key --key-id $keyname --profile ProfileName | jq -r '.KeyMetadata.Arn')
    echo "Key ARN is $key_arn"
    query_id=$(aws logs --profile ProfileName start-query --log-group-name $loggroupname --start-time 1655588640 --end-time 1659109342 --query-string "FIELD eventTime, userIdentity.userName, sourceIPAddress | SORT eventTime DESC | FILTER eventName = 'Decrypt' AND resources.0.ARN = '$key_arn'" | jq -r '.queryId')
    sleep 60
    echo "\e[96m$(aws logs --profile ProfileName get-query-results --query-id $query_id)\e[0m"
done

If you receive something like the following snippet for all the keys and you suspect some of them were used at least once during the timespan, try another log group. In my case, the log group that produced such results was disabled (checked in AWS GUI Management Console).

{
    "results": [],
    "statistics": {
        "recordsMatched": 0.0,
        "recordsScanned": 0.0,
        "bytesScanned": 0.0
    },
    "status": "Complete"
}

Another useful command to see view these events with fzf timestamps:

aws logs get-query-results --profile ProfileName --query-id "$query_id" | jq -c '.results[]' | fzf --multi --cycle --reverse --preview-window=right:80%:wrap --preview 'echo {} | jq .'

The following examples are limited to three events. You’ll need to remove --max-results 3 after testing it.

 aws cloudtrail lookup-events --profile ProfileName --max-results 3 | jq -c '.Events[]' | fzf --multi --cycle --reverse --preview-window=right:80%:wrap --preview 'echo {} | jq .' # --profile ProfileName is optional and is used in case of SSO auth only. 

# limit by UNIX timestamps
aws cloudtrail lookup-events --profile ProfileName --start-time 1655535200 --end-time 1655542800 | jq -c '.Events[]' | fzf --multi --cycle --reverse --preview-window=right:80%:wrap --preview 'echo {} | jq .'


# useful command to covert to UNIX timestamps
date -ju -f "%F %T" "2022-06-18 08:00:00" "+%s"

VPC Flow Logs

If these are enabled, you will be able to tell, who connected to what resource and when. Flow Logs are not enabled by default. It’s not the same as pcap file! It doesn’t provide full packet info. Just statistics, addresses, ports, result (accept/reject).

Logs are sent then either to the CloudWatch or S3 (depending on the configuration). An IAM role needs to be created prior to that. Here are the minimum requirements for the CloudWatch role: CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups, DescribeLogStreams need to beallowed for all resources. To be able to assume this role, VPC Flow Logs service needs to have its own policy where "service":"vpc-flow-logs.amazonaws.com" is set to allow AssumeRole action. See here for more details.

Log breakthrough. Looks like a simple string with a space as a delimiter. Each field has its place within the string.

Traffic logs between VPCs withing the same org/acc.

If these are enabled, you will be able to tell, who connected to what resource and when. Flow Logs are not enabled by default. AWS states that loggin rejected traffic is sufficient for bad actor detection. The recommended log retention period is 1 year. However, these logs can be archived and stored elsewhere where it’s cheaper.

It’s not the same as pcap file! It doesn’t provide full packet info. Just statistics, addresses, ports, result (accept/reject).

Logs are sent then either to the CloudWatch or S3 (depending on the configuration). Min delay between the capture and its copying to the above mentioned services is 1-6 minutes.

📝 https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/logging-and-events.html

📘 BTFM

Retrieve the logs:

aws ec2 describe-flow-logs # append --profile for SSO or make sure you have the correct creds in ~/.aws/config
aws logs filter-log-events --log-group-name <groupname>  # append --profile for SSO or make sure you have the correct creds in ~/.aws/config

Filter the logs - docs.

Resource Timeline

In AWS go to Config → Resources → <resource name> → Resource Timeline (in the upper right corner). Shows all the changed made to the resource (created, configuration changed, resource deleted).

IAM Activity

Two places to look at it.

  • Credentials report

iam/home#/credential_report

Contains the following fields:

  1. user
  2. arn
  3. user_creation_time
  4. password_enabled
  5. password_last_used
  6. password_last_changed
  7. password_next_rotation
  8. mfa_active
  9. access_key_1_active
  10. access_key_1_last_rotated
  11. access_key_1_last_used_date
  12. access_key_1_last_used_region
  13. access_key_1_last_used_service
  14. access_key_2_active
  15. access_key_2_last_rotated
  16. access_key_2_last_used_date
  17. access_key_2_last_used_region
  18. access_key_2_last_used_service
  19. cert_1_active
  20. cert_1_last_rotated
  21. cert_2_active
  22. cert_2_last_rotated

Access Advisor

Go to IAM -> username -> Access Advisor. Not every user might (and should) have access to this one. Access Advisor shows the services that this user can access and when those services were last accessed.

VPC Traffic Mirroring

A great idea to use. Deploying an EC2 with some IDS, mirror traffic there for detection. Filter for the most relevant stuff to pass it to SIEM?

VPN

If the organization uses a VPN in order to connect, an investigator will need the VPN logs as well (IPsec or OpenVPN).

Logs Archieved In S3

CloudTrail logs are not available forever. If the company backs up all events as they come and sends them to a S3 bucket. How to query them? To search for a particular string within the logs. For example, by event_id or username.

Bash script below will look through the files recursively and run grep against each. This is very time consuming, so you’d better narrow down the search and choose the keyword wisely.

⚠️ Make sure you’ve logged in the Log Archive first and you have set up the profile for SSO in ~/.aws/config.

keyword="bucket-name"

for file in $(aws s3 ls --profile ProfileName --recursive "s3://bucket-name/path/to/CloudTrail/logs/")
do 
    if echo $file | grep -q "unique string"; then
        echo "========================================"
        echo "Looking for keywords in the file \e[36m$file...\e[0m"
        echo ""
        result=$(aws s3 cp "s3://bucket-name/$file" --profile ProfileName - | gzip -d - )
        if echo $result | grep -q $keyword; then
            echo $result | jq -C .
            echo $result | jq -C . >> result.txt
        fi
        echo "========================================"
        
    fi
done

$(aws s3 ls --profile ProfileName --recursive "s3://bucket-name/path/to/CloudTrail/logs/") will return a table: time - bytes - path (each on a separate line). So, when iterating through the files in S3 buckets, you’ll get time after the first iteration, size in bytes after the second one and the last iteration will get you the path itself. For the script to work, we need path only, that’s why I added this line if echo $file | grep -q "bucket-name"; then. This command aws s3 cp "s3://bucket-name/$file" --profile Backup - | gzip -d - will copy the contents of the file to the stdout (-) and pipe to gzip (since log files are usually compressed) and send to stdout again (-). If the file has the $keyword (if echo $result | grep -q $keyword; then) I run jq -C . (coloured jq output) and also save the result in a text file (append).

CloudWatch

Similarly, session activity audit logs can be captured in S3 or CloudWatch logs.

SQS

This can be used for various things but one of the usages could be a log buffer.

📘 BTFM

aws sqs list-queues  --profile ProfileName
aws sqs send-message --queue-url <value> --message-body <value>
aws sqs receive-message --queue-url <value>

🧪 I would like to see if messages will disappear from the queue once receive-message was invoked.

Security Monkey

???

References

[1] IR in AWS