TIL: GitHubtop Real-time Activity Monitor and GoAccess Web Log Analyzer

ghtop - Real-time GitHub Activity Monitor

GitHub - nat/ghtop: See what’s happening on GitHub in real time

Terminal-based tool for monitoring GitHub activity in real-time:

What It Does:

• Live Activity: Real-time stream of GitHub events
• API Consumption: Helpful for using up GitHub API quota quickly
• Terminal Interface: htop-style display for GitHub events
• Event Filtering: Focus on specific types of GitHub activity

Key Features:

Event Types Monitored:

• Push Events: Code commits to repositories
• Pull Requests: PR creation, updates, merges
• Issues: Issue creation, comments, closures
• Releases: New releases and tags
• Forks: Repository forks
• Stars: Repository stars and watches

Display Information:

Use Cases:

API Quota Management:

• Testing: Quickly consume API rate limits for testing
• Development: Understand API usage patterns
• Monitoring: Track when rate limits reset
• Debugging: Test API error handling

Community Insights:

• Trending Activity: See what’s active on GitHub right now
• Development Patterns: Understand when developers are most active
• Project Monitoring: Watch for activity on specific projects
• Language Trends: Observe which languages are being used

Educational Value:

• GitHub API: Learn about GitHub’s event API structure
• Real-time Data: Understand streaming data concepts
• Rate Limiting: Experience API rate limiting firsthand

Installation and Usage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Installation
npm install -g ghtop

# Basic usage
ghtop

# With API token for higher rate limits
GITHUB_TOKEN=your_token ghtop

# Filter by event type
ghtop --filter=PushEvent

GoAccess - Visual Web Log Analyzer

GoAccess - Visual Web Log Analyzer

Real-time web log analyzer and interactive viewer:

Core Capabilities:

Log Format Support:

• Apache: Common and combined log formats
• Nginx: Standard and custom log formats
• IIS: Microsoft web server logs
• Amazon S3: S3 access logs
• CloudFlare: CDN access logs
• Custom Formats: Define your own log patterns

Real-time Analysis:

1
2
3
4
5
6
7
8

# Real-time monitoring
tail -f /var/log/nginx/access.log | goaccess -

# Generate HTML report
goaccess /var/log/nginx/access.log -o report.html

# Real-time HTML dashboard
goaccess /var/log/nginx/access.log -o report.html --real-time-html

Rich Analytics Dashboard:

Traffic Metrics:

• Unique Visitors: Daily unique IP addresses
• Requested Files: Most popular pages and resources
• Static Requests: CSS, JS, images analytics
• 404 Errors: Broken links and missing resources
• Hosts: Top visitor IP addresses and domains

Detailed Reports:

Advanced Features:

GeoIP Integration:

1
2
3
4
5

# Install GeoIP database
sudo apt-get install geoip-database

# Enable geographic reporting
goaccess --geoip-database=/usr/share/GeoIP/GeoIP.dat

Geographic Analytics:

• Countries: Visitor distribution by country
• Cities: Top cities accessing your site
• Maps: Visual geographic representation
• Timezone Analysis: Traffic patterns by timezone

Custom Log Formats:

1
2
3
4
5
6
7
8
9

# Define custom nginx format
goaccess --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u"' \
         --date-format='%d/%b/%Y' \
         --time-format='%T'

# Complex format with custom fields
goaccess --log-format='%h %^ %^ [%d:%t %^] "%r" %s %b "%R" "%u" %D %T' \
         --date-format='%d/%b/%Y' \
         --time-format='%T'

Interactive HTML Dashboard:

• Real-time Updates: Live data streaming
• Responsive Design: Mobile-friendly interface
• Interactive Charts: Clickable graphs and tables
• Export Options: PDF, CSV, JSON export capabilities

Performance Optimization:

Large Log Files:

1
2
3
4
5
6
7
8

# Use --no-global-config for better performance
goaccess --no-global-config /var/log/nginx/*.log

# Process compressed logs
zcat /var/log/nginx/*.gz | goaccess -

# Multiple log files
goaccess /var/log/nginx/access.log.* --log-format=COMBINED

Memory Management:

• Streaming Mode: Process logs without loading entirely into memory
• Incremental Updates: Add new log entries to existing reports
• Disk Storage: Store data structures on disk for large datasets

Integration and Automation:

Automated Reporting:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

#!/bin/bash
# Daily report generation
goaccess /var/log/nginx/access.log \
    --log-format=COMBINED \
    --output-format=html \
    --output=/var/www/html/stats/daily-$(date +%Y%m%d).html

# Email reports
goaccess /var/log/nginx/access.log \
    --log-format=COMBINED \
    --output-format=csv | \
    mail -s "Daily Website Stats" [email protected]

CI/CD Integration:

• Performance Monitoring: Track site performance over time
• Traffic Analysis: Understand user behavior patterns
• Security Monitoring: Detect unusual access patterns
• Capacity Planning: Analyze traffic growth trends

Configuration Options:

Color Schemes:

1
2
3
4
5

# Dark theme
goaccess --color-scheme=2

# Custom colors
goaccess --html-custom-css=styles.css

Filtering:

1
2
3
4
5
6
7
8

# Exclude specific IPs
goaccess --exclude-ip=192.168.1.1

# Include only specific paths
goaccess --ignore-panel=KEYPHRASES,REFERERS

# Date range filtering
goaccess --date-spec=hr  # Hourly breakdown

Both tools provide valuable insights into different aspects of software development and web operations - real-time GitHub activity monitoring and comprehensive web traffic analysis.