Postback Delivery is a web-based app that ingests, queues, and delivers incoming http requests. All of the code has been developed and maintained by Mark Wunsch.

1. Installation/Configuration
   • Technology Stack
   • Installation
   • Configuration
2. Usage
   • Data Flow
   • Sample Request
   • Sample Response
   • Test Case
3. Components
   • Ingestion Agent (PHP)
   • Delivery Queue (Redis)
   • Delivery Agent (GO)
4. Future Work
   • Planned Features
   • Wish List

Everything you should need to get this web app installed and functional on a new system.

1. Ubuntu
2. Apache2
3. PHP
4. phpredis
5. Redis
6. Redigo
7. GO

The following commands were used to install the necessary software on an already existing instance of Ubuntu:

GCC

apt-get install gcc

Apache2

apt-get install apache2

PHP

apt-get install php5 php5-dev libapache2-mod-php5

Redis

wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
cd deps
make hiredis jemalloc linenoise lua
cd ..
make && make install
cd utils
./install_server.sh (this will cause redis-server to start on boot)

phpredis (after downloading source from github)

phpize
./configure
make && make install

GO (might only work on newer versions of Ubuntu)

apt-get install golang

Redigo

go get github.com/garyburd/redigo/redis

Add 'ingest.php' to /etc/apache2/mods-enabled/dir.conf

Run "a2enmod rewrite" command to enable mod_rewrite

Add the following to .htaccess file in site's directory root:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond ${REQUEST_FILENAME} !-f
RewriteCond ${REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /ingest.php [L]
</IfModule>

In /etc/apache2/apache2.conf, set "AllowOverride" to "all"

Update-rc.d apache2 defaults (this will make apache2 start on boot)

Add "extension=redis.so" to php.ini file

Run the following commands:

export GOPATH=/root/go
export GOBIN=$GOPATH/bin

These variables can/should be added to ~/.bashrc

Add "/go/root/bin/DeliveryAgent &" to /etc/rc.local (Note: there is almost certainly a better way to run DeliveryAgent as a daemon, but this will for now)

1. Web request to 162.243.19.49/ingest.php or 162.243.19.49/i >
2. Ingestion Agent (php) >
3. Delivery Queue (redis) >
4. Delivery Agent (GO) >
5. Web response >
6. Response logged in /var/log/DeliveryAgent/postback_log.txt and/or /var/log/DeliveryAgent/error.txt (Note: log names are congfigurable in DeliveryAgent.go)

(POST) http://{server_ip}/ingest.php
	(RAW POST DATA) {  
	  "endpoint":{  
	    "method":"GET",
	    "url":"http://sample_domain_endpoint.com/data?key={key}&value={value}&foo={bar}"
	  },
	  "data":[  
	    {  
	      "key":"Azureus",
	      "value":"Dendrobates"
	    },
	    {  
	      "key":"Phyllobates",
	      "value":"Terribilis"
	    }
	  ]
	}

http://sample_domain_endpoint.com/data?key=Phyllobates&value=Terribilis&foo=

• 162.243.19.49/index.html can be used to execute a very simple test of the Postback Delivery.

The Ingestion Agent consists of the following components:

• ingest.php
  • This file accepts the incoming web request and intializes a new endpointRequest with the contents of the request.
• endpointRequest.php
  • This class is the main worker for the ingestion agent. It creates a new dataRequest for each individual data item and handles the interaction with Redis.
• dataRequest.php
  • This helper class keeps track of each individual data point from the original web request. One postback will be created for each dataRequest.

The delivery queue is created and maintained in an instance of Redis. It contains the following data structures:

• Pending (List of UUIDs)
  • Maintains a list of all UUIDs, each of which corresponds with a unique postback request.
  • Utilized as a First-In, First-Out (FIFO) list though the use of "LPUSH" for adding and "RPOP" for retrieving.
• Values (Hash)
  • UUID from Pending list can be hashed to retrieve that URL of the postback request.
  • "UUID:method" can be hashed to retrieve the method of the postback request.
• Stats (Hash)
  • All time values stored in Stats are in milliseconds in Unix time.
  • "UUID:start" can be hashed to retrieve the time the postback request was received.
• Working (Sorted Set)
  • When the delivery agent begins processing a postback request, that UUID is added to this set with the time it began processing.
  • If a postback request is succesfully handled by the delivery agent, its UUID will be removed from Working
  • If the delivery agent fails during the processing of a request, its UUID will remain on this list to be retrieved and reprocessed.
• Delayed (Sorted Set) NOT YET IMPLEMENTED
  • This sorted set can be used to handle postback requests that are to be delivered after some configurable delay.
  • Moving a UUID from this set to Pending would begin the processing of that request.

The Delivery Agent is handled by DeliveryAgent.go (found at /root/go/src/DeliveryAgent/DeliveryAgent.go)

• Infinite loop that constantly attempts to RPOP from a Redis list called Pending.
• When a new postback request is found in the Redis data sets, DeliveryAgent.go executes an HTTP "GET" and logs the results.
• Constants at the top of the file can be used to define the Redis connection, Redis data sets, and log file locations.
• Request Time, Delivery Time, and Response Time are all logged in UTC Unix time (milliseconds)
  • Request Time comes from the Ingestion Agent and represents the time the response was received by ingest.php
  • Delivery Time comes from the Delivery Agent and represents the time that DeliveryAgent.go begins handling a postback request.
  • Response Time comes from the Delivery Agent and represents the time that the HTTP "GET" response was received.

• More comprehensive testing suite (currently just using hard-coded values in ingest_test.js)
• Accept JSON encoded web requests (application/x-www-form-urlencoded currently used for simplicity and speed)
• GO application to handle orphaned requests in Working Redis data structure.
• Updating Delivery Agent and Ingestion Agent to handle "POST" requests as well
• Add configurable delay to postback requests
• Add configurable number of retry attempts on failed requests
• Run DeliveryAgent as a daemon

• Visualization of postbacks in real-time
• Application monitoring/profiling