Skip to content

Latest commit

 

History

History
85 lines (81 loc) · 3.65 KB

readme.md

File metadata and controls

85 lines (81 loc) · 3.65 KB
Raw

This does NOT work

this code is alpha at best, it does not work, gives loads of PHP errors. Just wait until it is finished, ok?

Install requirements

  • Install Ubuntu 16.04 DB with Elasticsearch and Kibana

  • Update Elasticsearch config for higher load:

    thread_pool:
    bulk:
        queue_size: 1000
    search:
        queue_size: 5000
    index:
        queue_size: 5000

    Note: Getting errors like below are an indication that the index.queue_size is not sufficiant:

    WEBHOOK An error occurred while handling │[2018-05-07 17:02:21] lumen.INFO: JOB: c01b2ca4-d179-49a1-bf79-5a5b9377371a WEBHOOK TI-REPORT saved into database : {"
this job, stack trace: {"error":{"root_cause":[{"type":"es_rejected_execution_exception","reason":"rejected execution │_index":"threat_indicators","_type":"threat_indicators","_id":"1563070557139172","_version":2,"result":"noop","_shards
of org.elasticsearch.transport.TcpTransport$RequestHandler@21b6639a

  • Install Ubuntu 16.04 WEB with following packages:

    apt-get install php7.0-dev php7.0-cli php7.0-zip php7.0-json php-pear php7.0-mysql 
apt-get install composer libapache2-mod-php7.0 php7.0-mcrypt php7.0-mbstring whois
apt-get install apache2 pwgen beanstalkd git php7.0-curl php7.0-intl

  • Install ZMQ and ZMQ-PHP, and enable the ZMQ php module

  • Enable Apache modules and disable libapache2, its a fallback as we use fpm

    a2enmod headers
a2enmod rewrite
a2enmod ssl
a2enmod proxy
a2enmod proxy-http
a2enmod remoteip
a2dismod php7.0

  • Copy the systemd file, update the hostname and enable workers (minimal 60 required!):

    cp contrib/aite-worker@.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable aite-worker@{0..100}.service
systemctl daemon-reload
systemctl start aite-worker@{0..100}.service

  • Update /etc/default/beanstalkd:

    BEANSTALKD_LISTEN_ADDR=0.0.0.0
BEANSTALKD_LISTEN_PORT=11300
BEANSTALKD_EXTRA="-b /var/lib/beanstalkd -z 524280"

  • YOU MUST Set a 32byte APP_KEY

  • You MUST enable SSL on your endpoint (at apache here, or at haproxy)

  • You SHOULD use haproxy with 3 backend WEB nodes

  • You SHOULD use ES with 2 copies (its prolly hardcoded at this moment :>)

  • You SHOULD tune apache to allow MASSSIVE updating from remote endpoints (!)

  • You SHOULD tune sysctl while your at it

  • Restart Apache2 after enabling modules and/or changing PHP settings!

Get Facebook access

  • sign up with facebook with a PERSONAL ACCOUNT (!)
  • visit https://developers.facebook.com
  • create application
  • collect the application ID from the newly generated app (put into ENV)
  • open the app dashboard by clicking the the app name = In the left menu, open settings->basic. Add a privacy policy (required!)
  • In the upper richt click the status towards LIVE
  • In the left menu click op the + after PRODUCTS and select webhook, and add it
  • The reporting URL will be vhost/get_report
  • The token will be the code you have in ENV file.
  • Collect the application 'App Token' from: https://developers.facebook.com/tools/accesstoken (put into ENV)
  • Visit https://graph.facebook.com/threat_exchange_members?access_token=APP_TOKEN_HERE and check for errors
  • run: php artisan threatex:subscribe
  • run: php artisan threatex:subscriptions
  • The last command should list a few feeds that are enabled. If not then your screwed (todo docs)
  • Go to settings->basic and get the application secret key, put in env as TI_APPLICATION_SECRET

Todo

  • move handler into a queue for asym with error handling
  • add more logging
  • collect alerts from facebook app
  • make CLI commands/autodetect to manage subscriptions