Image Info:



Cloud Image ID Source image
aws ami-046efcde04f30f4f7 ami-0b898040803850657

Kernel Info:



Processor architecture OS name OS release
x86_64 Linux 4.19.72-25.58.amzn2.x86_64

Raw package lists:



Installations:


yum Updates


Name Info
security --

yum Group Packages


Name Info
Development tools Required for R installations

yum Repositories


Name Info
datadog --
epel-release --
mysql56community --

yum Packages


Name Version Info
htop 1.0.1 Cluster Management
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm Required for xmlstarlet
zlib-devel 1.2.8 Cluster Management
openssl-devel 1.0.2k Required for python cyptography package installed in Hustler
readline-devel 6.2 Cluster Management
mysql 5.5 Required for Cloudman
collectl 3.6.7 Cluster Management
snappy 1.0.5 required for Hadoop2
snappy-devel 1.0.5 required for Hadoop2
git 2.14.5 --
libcurl-devel 7.61.1 required for R installations and needed to compile nginx
libffi-devel 3.0.13 Required for python cyptography package installed in Hustler
jq 1.5 used to parse json strings. SSL certificate generation uses it to extract certificate from the json response of opsapi/v1/ca/sign
xterm 253 resize window in local machine when accessing a remote machine
libpng-devel 1.2.49 Required for pandas
gcc-c++ 4.8.3 Cluster Management
cyrus-sasl-devel.x86_64 2.1.23 Support MVP of superset(caravel)
kernel-headers 4.14.123 For every release signoff, lets start publishing the kernel version, security update and other critical version information
ganglia 3.7.2 Cluster Monitoring
ganglia-gmond 3.7.2 Cluster Monitoring
ganglia-gmetad 3.7.2 Cluster Monitoring
ganglia-web 3.7.1 Cluster Monitoring
boost-devel 1.53.0 Cluster Management
boost-python 1.53.0 Cluster Management
boost-program-options 1.53.0 Cluster Management
expat-devel 2.1.0 Cluster Management
puppet 3.6.2 Cluster Management
util-linux 2.30.2 Cluster Management
e2fsprogs 1.42.9 Cluster Management
aws-apitools-ec2 1.7.3.0 Cluster Management
gpgme 1.3.2 Cluster Management
gpgme-devel 1.3.2 Cluster Management
rrdtool 1.4.8 Cluster Management
chkconfig 1.3.49.3 --
libjpeg-turbo-devel 1.2.90 Required for PIL egg used by Pinterest
libpng-devel 1.5.13 Required for PIL egg used by Pinterest
freetype-devel 2.4.11 Required for PIL egg used by Pinterest
python-nose 1.3.7 Required for scipy
gcc-gfortran 7.3.1 Required for scipy and R package installations
blas-devel 3.4.2 Required for scipy
lapack-devel 3.4.2 Required for scipy
atlas-devel 3.10.1 Required for scipy
libxml2-devel 2.9.1 Required for lxml
libxslt-devel 1.1.28 Required for lxml
openmpi-devel 2.1.1 Required for mpi4py
openmpi 2.1.1 Required for mpi4py
erlang R14B Required for rabbitmq
mysql-community-common 5.6.* AL2 base bake failing due to mysql dependent package issue
mysql-community-server 5.6.* --
mysql-community-client 5.6.* --
mysql-community-devel 5.6.* --
postgresql 9.2.23 The default cluster datastore for Airflow Clusters has changed from MySQL to Postgresql
postgresql-server latest install postgres server on AL2
postgresql-contrib latest install postgres server on AL2
nfs-utils 1.3.0 Install nfs-utils and cachefilesd in the AMI so users can use Amazon EFS for virtualenvs and other data.
cachefilesd 0.10.5 Install nfs-utils and cachefilesd in the AMI so users can use Amazon EFS for virtualenvs and other data.
chrony 3.2 Incorrectly configured DNS entries can cause ntpd to hang for several minutes. To avoid this we should replace ntpd on our cluster AMI with Amazon's chrony
nginx 1.10.3 As of now we are bringing up only 1 Jeeves process in cluster master. we might want to launch multiple (say 4) and use nginx for load-balancing.
parallel 20160722 Cluster Management
inotify-tools 3.14 Cluster start time improvements
libuuid-devel 2.23.2 Cluster Management
device-mapper-devel 1.02.135 Cluster Management
jdk1.8 1.8.0_192 --
lz4 r131 Needed for tar compression
libicu-devel.x86_64 50.1.2 needed for stringi installation
libXdmcp.x86_64 1.1.1 needed dependency for R Library (knitr), R plots rendering issues when package management is enabled are fixed
datadog-agent 6.14.0 Monitoring and analysis of performance data
docker 18.03.1ce The previous docker version had security vulnerabilities, which could allow malicious containers to gain root-level privileges on the host. To resolve security vulnerabilities of the previous docker version, the latest patch version of docker-18.06.1ce is installed.
docker-compose 1.18.0 Required for docker
haproxy 1.5.2 With this feature Qubole has added haproxy in cluster to load balance between multiple connections to metastore from cluster (in case of 'Qubole managed metastore')
python34 3.4.10 enable the consumption of AL2 images in clusters.
expect 5.44.1.15 Required for KNOX startup
python36 3.6.8 Provide Python3.6 in control/data plane AMIs for Cloudman and Asterix
python36-devel 3.6.8 Provide Python3.6 in control/data plane AMIs for Cloudman and Asterix
xmlstarlet 1.3.1 This will help in playing around with xml outputs viz. that of monit.
nvme-cli 0.7 nvme-cli required to avoid cluster bringup failure for instances with nvme volumes.

R Packages


Name Version Info
htmltools 0.3.6 Packages required for Data Science
knitr 1.25 Packages required for Data Science
devtools 2.2.0 Packages required for Data Science
evaluate 0.14 Packages required for Data Science
googleVis 0.6.4 Packages required for Data Science
base64enc 0.1-3 Packages required for Data Science
lazy 1.2-16 Packages required for Data Science
IRkernel/repr from GitHub Packages required for Data Science
mplot 1.0.3 Packages required for Data Science
googleVis 0.6.4 Packages required for Data Science
ramnathv/rCharts from GitHub Packages required for Data Science

Other installations


Name Version Info
rabbitmq-server 3.5.6 Required to install Airflow
parted 3.2 Cluster Management
pip 2.6 --
Anaconda2 4.2.0 Add Anaconda to AMI
scala ['2.10.4', '2.11.7'] Required for Spark
redis 3.2.4 Create a key-value database as a cache
autossh 1.4e Monitoring SSH sessions
monit 5.25.2 Monit daemon in base image
ruby 2.1.6 Deploy ruby using RVM
gem 1.6.2 Deploy ruby using RVM
cuda 4.0.0 Install cuda on base AMI

python Packages


Package Version
awscli 1.16.224
s3cmd 2.0.2
virtualenv 16.7.3

Amazon Linux extras


Name Version Info
R3.4 latest Required for Data Science
kernel-ng latest Install latest kernel in AL2
lustre2.10 latest --

python Virtual Environments


Name Info
hustler Python virutal environment for hustler
    Package Version
    argparse 1.2.1
    backports.ssl-match-hostname 3.4.0.2
    begins 0.9
    boto 2.38.0
    boto3 1.9.130
    botocore 1.12.130
    chardet 3.0.4
    cryptography 1.9
    datadog 0.20.0
    decorator 4.0.9
    docutils 0.12
    ecdsa 0.11
    enum34 1.1.5
    freezegun 0.3.7
    futures 3.0.5
    idna 2.7
    ipaddress 1.0.16
    Jinja2 2.8
    jmespath 0.9.0
    MarkupSafe 0.23
    MySQL-python 1.2.5
    ordereddict 1.1
    paramiko 2.0.3
    pssh 2.3.1
    psutil 4.3.1
    pyasn1 0.1.9
    pycparser 2.14
    pycurl 7.19.0
    python-dateutil 2.5.3
    python-magic 0.4.11
    PyYAML 3.11
    recordclass 0.4.1
    requests 2.19.1
    rsa 3.3
    s3cmd 1.5.2
    simplejson 3.3.0
    six 1.8.0
    PySocks 1.5.7
    total-ordering 0.1.0
    urllib3 1.23
    workerpool 0.9.4
    inotify 0.2.9
    IPy 1.0
Name Info
cloudman Python virutal environment for cloudman
    Package Version
    boto 2.49.0
    adal 0.4.7
    appdirs 1.4.0
    asn1crypto 0.22.0
    boto3 1.4.4
    botocore 1.5.95
    certifi 2017.4.17
    cffi 1.9.1
    chardet 3.0.4
    click 6.6
    configparser 3.5.0
    cryptography 2.1.3
    datadog 0.20.0
    decorator 4.0.11
    docutils 0.13.1
    freezegun 0.3.7
    future 0.16.0
    httplib2 0.11.3
    idna 2.5
    isodate 0.5.4
    Jinja2 2.7
    jmespath 0.9.1
    keyring 10.2
    lxml 3.6.4
    MarkupSafe 1.0
    msrest 0.4.25
    msrestazure 0.4.20
    mysqlclient 1.3.9
    oauth2client 4.1.2
    oauthlib 2.0.1
    packaging 16.8
    paramiko 2.0.3
    pssh 2.3.1
    psutil 5.0.1
    pyasn1 0.4.2
    pyasn1-modules 0.2.2
    pycparser 2.17
    PyJWT 1.4.2
    pyOpenSSL 17.4.0
    pyparsing 2.1.10
    PySocks 1.5.7
    python-dateutil 2.5.3
    python-magic 0.4.15
    pytz 2016.10
    PyYAML 3.12
    recordclass 0.4.1
    requests 2.18.4
    requests-oauthlib 0.8.0
    rsa 3.4.2
    s3cmd 2.0.2
    s3transfer 0.1.10
    SecretStorage 2.3.1
    sh 1.12.13
    simplejson 3.16.0
    six 1.11.0
    typing 3.5.3.0
    uritemplate 3.0.0
    urllib3 1.21.1
    responses 0.3.0
    diskcache 3.0.6
    IPy 1.0
Name Info
python27 Virtual environment with python 2.7
    Package Version
    awscli 1.16.60
    alembic 0.8.6
    amqp 1.4.9
    anyjson 0.3.3
    Babel 1.3
    backports.ssl-match-hostname 3.5.0.1
    billiard 3.3.0.23
    boto 2.40.0
    bleach 2.0.0
    celery 3.1.23
    certifi 2016.2.28
    chartkick 0.4.2
    croniter 0.3.12
    dill 0.2.5
    Flask 0.10.1
    Flask-Admin 1.4.0
    Flask-Cache 0.13.1
    Flask-Login 0.2.11
    Flask-WTF 0.14
    flower 0.9.2
    future 0.15.2
    futures 3.0.5
    gunicorn 19.3.0
    inflection 0.3.1
    itsdangerous 0.24
    Jinja2 2.8
    kombu 3.0.35
    Mako 1.0.4
    Markdown 2.6.6
    MarkupSafe 0.23
    numpy 1.11.1rc1
    pandas 0.18.1
    prometheus_client 0.4.2
    Pygments 2.1.3
    psycopg2 2.7.1
    python-dateutil 2.5.3
    python-editor 1.0.1
    pytz 2016.4
    qds-sdk 1.10.0
    requests 2.10.0
    setproctitle 1.1.10
    six 1.10.0
    SQLAlchemy 1.1.0b1
    thrift 0.9.3
    tornado 4.2
    urllib3 1.16
    Werkzeug 0.11.10
    wheel 0.24.0
    WTForms 2.1
    lxml 2.3
    bs4 0.0.1
    awscli 1.16.224
    s3cmd 2.0.2
    virtualenv 16.7.3

Qubole Packages


Name Status Engine
hustler Active -
hive13 Deprecated -
pig Deprecating Soon 0.11
pig Deprecating Soon 0.15
pig Deprecating Soon 0.17
hadoop2 Active 2.6.0
hadoop2 Active 2.8.1
hadoop2 Active 3.1.0
hive2 Active -
hive1_2 Active 1.2
hive1_2 Active 1.2.1
hive1_2 Active 2.1.1
hive1_2 Active 2.3
hive1_2 Active 3.1.1
hbase Active -
presto Active 0.157
presto Active 0.180
presto Active 0.193
presto Active 0.208
spark_common Active -
spark Active 1.5.1
spark Active 1.6.0
spark Active 1.6.1
spark Active 1.6.2
spark Active 2.0.0
spark Active 2.0.2
spark Active 2.1.0
spark Active 2.1.1
spark Active 2.2.0
spark Active 2.2.1
spark Active 2.3.1
spark Active 2.3.2
spark Active 2.4.0
spark Active 2.4.3
zeppelin Active 1.5
zeppelin Active 1.6
zeppelin Active 2.0
zeppelin Active 2.1
zeppelin Active 2.2
zeppelin Active 2.3
zeppelin Active 2.4
jupyterlab Active -
tez Active 0.7.0
tez Active 0.8.4
tez Active 0.9.1
sqoop_h2 Active 1.4.6
sqoop_h2 Active 1.4.7
airflow Active 1.10.0
airflow Active 1.10.2.QDS
airflow Active 1.7.0
airflow Active 1.8.2
metricsd Active -
dr_elephant Active -
jeeves Active -
client_wrappers Active -
quboled Active -
knox Active -
prometheus Active -
cloudman Active -
megamind Active -
qds_ml Active -
ml_metastore Active -
logan Active -
rstudio_utils Active fuse_s3fs
rstudio_utils Active rsd
rubix Active -
file_merge_utility Active -
sparklens Active -