Image Info:



Cloud Image type Builder type Image ID Source image Base image
aws release_image aws-al2-hvm ami-046efcde04f30f4f7 ami-0b898040803850657 ami-0de6833c05cb429af

Kernel Info:



Processor architecture OS name OS release
x86_64 Linux 4.19.72-25.58.amzn2.x86_64

Raw package lists:



Installations:


yum Updates


Name Author Project Info
security -- TOOLS --

yum Group Packages


Name Author Project Info
Development tools -- TOOLS Required for R installations

yum Repositories


Name Author Project Info
datadog -- TOOLS --
epel-release -- TOOLS --
mysql56community -- TOOLS --

yum Packages


Name Version Author Project JIRA Info
htop 1.0.1 Sameer Kedia ACM ACM-284 Cluster Management
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm Anmol Porwal TOOLS TOOLS-1396 Required for xmlstarlet
zlib-devel 1.2.8 Sameer Kedia ACM ACM-284 Cluster Management
openssl-devel 1.0.2k Sameer Kedia ACM ACM-284 Required for python cyptography package installed in Hustler
readline-devel 6.2 Sameer Kedia ACM ACM-284 Cluster Management
mysql 5.5 Sameer Kedia ACM ACM-284 Required for Cloudman
collectl 3.6.7 Sameer Kedia ACM ACM-284 Cluster Management
snappy 1.0.5 Sameer Kedia ACM ACM-284 required for Hadoop2
snappy-devel 1.0.5 Sameer Kedia ACM ACM-284 required for Hadoop2
git 2.14.5 -- TOOLS No JIRA --
libcurl-devel 7.61.1 Monika Khandelwal ACM SPAR-1466 required for R installations and needed to compile nginx
libffi-devel 3.0.13 Sameer Kedia ACM ACM-284 Required for python cyptography package installed in Hustler
jq 1.5 Sameer Kedia ACM ACM-284 used to parse json strings. SSL certificate generation uses it to extract certificate from the json response of opsapi/v1/ca/sign
xterm 253 Sameer Kedia ACM ACM-284 resize window in local machine when accessing a remote machine
libpng-devel 1.2.49 Somya Kumar DL TOOLS-563 Required for pandas
gcc-c++ 4.8.3 Sameer Kedia ACM ACM-284 Cluster Management
cyrus-sasl-devel.x86_64 2.1.23 Sriram Ganesan ACM ACM-1165 Support MVP of superset(caravel)
kernel-headers 4.14.123 Rajendra Kumar TOOLS TOOLS-596-2 For every release signoff, lets start publishing the kernel version, security update and other critical version information
ganglia 3.7.2 Shashank K S ACM TOOLS-1105 Cluster Monitoring
ganglia-gmond 3.7.2 Shashank K S ACM TOOLS-1105 Cluster Monitoring
ganglia-gmetad 3.7.2 Shashank K S ACM TOOLS-1105 Cluster Monitoring
ganglia-web 3.7.1 Shashank K S ACM TOOLS-1105 Cluster Monitoring
boost-devel 1.53.0 Sameer Kedia ACM ACM-284 Cluster Management
boost-python 1.53.0 Sameer Kedia ACM ACM-284 Cluster Management
boost-program-options 1.53.0 Sameer Kedia ACM ACM-284 Cluster Management
expat-devel 2.1.0 Sameer Kedia ACM ACM-284 Cluster Management
puppet 3.6.2 Sameer Kedia ACM ACM-284 Cluster Management
util-linux 2.30.2 Sameer Kedia ACM ACM-284 Cluster Management
e2fsprogs 1.42.9 Sameer Kedia ACM ACM-284 Cluster Management
aws-apitools-ec2 1.7.3.0 Sameer Kedia ACM ACM-284 Cluster Management
gpgme 1.3.2 Sameer Kedia ACM ACM-284 Cluster Management
gpgme-devel 1.3.2 Sameer Kedia ACM ACM-284 Cluster Management
rrdtool 1.4.8 Sameer Kedia ACM ACM-284 Cluster Management
chkconfig 1.3.49.3 -- TOOLS No JIRA --
libjpeg-turbo-devel 1.2.90 Sameer Kedia ACM ACM-284 Required for PIL egg used by Pinterest
libpng-devel 1.5.13 Sameer Kedia ACM ACM-284 Required for PIL egg used by Pinterest
freetype-devel 2.4.11 Sameer Kedia ACM ACM-284 Required for PIL egg used by Pinterest
python-nose 1.3.7 Sameer Kedia ACM ACM-284 Required for scipy
gcc-gfortran 7.3.1 Sameer Kedia ACM ACM-284 Required for scipy and R package installations
blas-devel 3.4.2 Sameer Kedia ACM ACM-284 Required for scipy
lapack-devel 3.4.2 Sameer Kedia ACM ACM-284 Required for scipy
atlas-devel 3.10.1 Sameer Kedia ACM ACM-284 Required for scipy
libxml2-devel 2.9.1 Sameer Kedia ACM ACM-284 Required for lxml
libxslt-devel 1.1.28 Sameer Kedia ACM ACM-284 Required for lxml
openmpi-devel 2.1.1 Sameer Kedia ACM ACM-284 Required for mpi4py
openmpi 2.1.1 Sameer Kedia ACM ACM-284 Required for mpi4py
erlang R14B Yogesh Garg INFRA QBOL-4925 Required for rabbitmq
mysql-community-common 5.6.* Sanket Singhal ACM TOOLS-1254 AL2 base bake failing due to mysql dependent package issue
mysql-community-server 5.6.* Sanket Singhal ACM No JIRA --
mysql-community-client 5.6.* Sanket Singhal ACM No JIRA --
mysql-community-devel 5.6.* Sanket Singhal ACM No JIRA --
postgresql 9.2.23 Joy Lal Chattaraj Middleware AIR-277 The default cluster datastore for Airflow Clusters has changed from MySQL to Postgresql
postgresql-server latest Anmol Porwal TOOLS TOOLS-1664 install postgres server on AL2
postgresql-contrib latest Anmol Porwal TOOLS TOOLS-1664 install postgres server on AL2
nfs-utils 1.3.0 Hariharan Iyer Hadoop HADTWO-1087 Install nfs-utils and cachefilesd in the AMI so users can use Amazon EFS for virtualenvs and other data.
cachefilesd 0.10.5 Hariharan Iyer Hadoop HADTWO-1087 Install nfs-utils and cachefilesd in the AMI so users can use Amazon EFS for virtualenvs and other data.
chrony 3.2 Shashank K S HADOOP TOOLS-530 Incorrectly configured DNS entries can cause ntpd to hang for several minutes. To avoid this we should replace ntpd on our cluster AMI with Amazon's chrony
nginx 1.10.3 Kartik Borkar INFRA INFRA-636 As of now we are bringing up only 1 Jeeves process in cluster master. we might want to launch multiple (say 4) and use nginx for load-balancing.
parallel 20160722 Sameer Kedia ACM ACM-284 Cluster Management
inotify-tools 3.14 Hariharan Iyer ACM ACM-1498 Cluster start time improvements
libuuid-devel 2.23.2 Sameer Kedia ACM ACM-284 Cluster Management
device-mapper-devel 1.02.135 Sameer Kedia ACM ACM-284 Cluster Management
jdk1.8 1.8.0_192 -- TOOLS No JIRA --
lz4 r131 Divyanshu Rajan Zeppelin QWZ-301 Needed for tar compression
libicu-devel.x86_64 50.1.2 Divyanshu Rajan Zeppelin TOOLS-497 needed for stringi installation
libXdmcp.x86_64 1.1.1 Divyanshu Rajan Zeppelin ZEP-1800 needed dependency for R Library (knitr), R plots rendering issues when package management is enabled are fixed
datadog-agent 6.14.0 Nagendra Varma ACM TOOLS-154 Monitoring and analysis of performance data
docker 18.03.1ce karuppayya SPARK TOOLS-1139 The previous docker version had security vulnerabilities, which could allow malicious containers to gain root-level privileges on the host. To resolve security vulnerabilities of the previous docker version, the latest patch version of docker-18.06.1ce is installed.
docker-compose 1.18.0 karuppayya SPARK TOOLS-1139 Required for docker
haproxy 1.5.2 Harsh Desai INFRA INFRA-603-ami With this feature Qubole has added haproxy in cluster to load balance between multiple connections to metastore from cluster (in case of 'Qubole managed metastore')
python34 3.4.10 Sanket Singhal ACM ACM-3953 enable the consumption of AL2 images in clusters.
expect 5.44.1.15 Vaibhav Beriwala KNOX KNOX-27 Required for KNOX startup
python36 3.6.8 Manjunath Julpi TOOLS TOOLS-1060 Provide Python3.6 in control/data plane AMIs for Cloudman and Asterix
python36-devel 3.6.8 Manjunath Julpi TOOLS TOOLS-1060 Provide Python3.6 in control/data plane AMIs for Cloudman and Asterix
xmlstarlet 1.3.1 Anmol Porwal TOOLS TOOLS-1396 This will help in playing around with xml outputs viz. that of monit.
nvme-cli 0.7 Manav Sharma ACM ACM-5407 nvme-cli required to avoid cluster bringup failure for instances with nvme volumes.

R Packages


Name Version Author Project JIRA Info
htmltools 0.3.6 Somya Kumar DS DS Packages required for Data Science
knitr 1.25 Somya Kumar DS DS Packages required for Data Science
devtools 2.2.0 Somya Kumar DS DS Packages required for Data Science
evaluate 0.14 Somya Kumar DS DS Packages required for Data Science
googleVis 0.6.4 Somya Kumar DS DS Packages required for Data Science
base64enc 0.1-3 Somya Kumar DS DS Packages required for Data Science
lazy 1.2-16 Somya Kumar DS DS Packages required for Data Science
IRkernel/repr from GitHub Somya Kumar DS DS Packages required for Data Science
mplot 1.0.3 Somya Kumar DS DS Packages required for Data Science
googleVis 0.6.4 Somya Kumar DS DS Packages required for Data Science
ramnathv/rCharts from GitHub Somya Kumar DS DS Packages required for Data Science

Packages downloaded via wget


Name Version Author Project JIRA Info
rabbitmq-server 3.5.6 Yogesh Garg INFRA QBOL-4925 Required to install Airflow
parted 3.2 Sameer Kedia ACM ACM-284 Cluster Management
pip 2.6 -- TOOLS No JIRA --
Anaconda2 4.2.0 Divyanshu Rajan DL QWZ-201 Add Anaconda to AMI
scala ['2.10.4', '2.11.7'] Monika Khandelwal DS ACM-284 Required for Spark
redis 3.2.4 Abhishek Somani PRESTO PER-67 Create a key-value database as a cache
autossh 1.4e Harsh Desai INFRA INFRA-473 Monitoring SSH sessions
monit 5.25.2 Vaibhav Beriwala Hadoop HADTWO-1217 Monit daemon in base image

python Packages


Package Version
awscli 1.16.224
s3cmd 2.0.2
virtualenv 16.7.3

Amazon Linux extras


Name Version Author Project JIRA Info
R3.4 latest Monika Khandelwal DS No JIRA Required for Data Science
kernel-ng latest Anmol Porwal TOOLS TOOLS-1649 Install latest kernel in AL2
lustre2.10 latest Vaibhav Beriwala Hadoop HADTWO-1911 --

Installations via shell scripts


Package Version Author Project JIRA Info
ruby 2.1.6 Vikram Agrawal Qubole Eng No JIRA Deploy ruby using RVM
gem 1.6.2 Vikram Agrawal Qubole Eng No JIRA Deploy ruby using RVM
cuda 4.0.0 Somya Kumar Qubole Eng QWZ-190 Install cuda on base AMI

python Virtual Environments


Name Author Project Info
hustler Sameer Kedia ACM Python virutal environment for hustler
    Package Version
    argparse 1.2.1
    backports.ssl-match-hostname 3.4.0.2
    begins 0.9
    boto 2.38.0
    boto3 1.9.130
    botocore 1.12.130
    chardet 3.0.4
    cryptography 1.9
    datadog 0.20.0
    decorator 4.0.9
    docutils 0.12
    ecdsa 0.11
    enum34 1.1.5
    freezegun 0.3.7
    futures 3.0.5
    idna 2.7
    ipaddress 1.0.16
    Jinja2 2.8
    jmespath 0.9.0
    MarkupSafe 0.23
    MySQL-python 1.2.5
    ordereddict 1.1
    paramiko 2.0.3
    pssh 2.3.1
    psutil 4.3.1
    pyasn1 0.1.9
    pycparser 2.14
    pycurl 7.19.0
    python-dateutil 2.5.3
    python-magic 0.4.11
    PyYAML 3.11
    recordclass 0.4.1
    requests 2.19.1
    rsa 3.3
    s3cmd 1.5.2
    simplejson 3.3.0
    six 1.8.0
    PySocks 1.5.7
    total-ordering 0.1.0
    urllib3 1.23
    workerpool 0.9.4
    inotify 0.2.9
    IPy 1.0
Name Author Project Info
cloudman Sameer Kedia ACM Python virutal environment for cloudman
    Package Version
    boto 2.49.0
    adal 0.4.7
    appdirs 1.4.0
    asn1crypto 0.22.0
    boto3 1.4.4
    botocore 1.5.95
    certifi 2017.4.17
    cffi 1.9.1
    chardet 3.0.4
    click 6.6
    configparser 3.5.0
    cryptography 2.1.3
    datadog 0.20.0
    decorator 4.0.11
    docutils 0.13.1
    freezegun 0.3.7
    future 0.16.0
    httplib2 0.11.3
    idna 2.5
    isodate 0.5.4
    Jinja2 2.7
    jmespath 0.9.1
    keyring 10.2
    lxml 3.6.4
    MarkupSafe 1.0
    msrest 0.4.25
    msrestazure 0.4.20
    mysqlclient 1.3.9
    oauth2client 4.1.2
    oauthlib 2.0.1
    packaging 16.8
    paramiko 2.0.3
    pssh 2.3.1
    psutil 5.0.1
    pyasn1 0.4.2
    pyasn1-modules 0.2.2
    pycparser 2.17
    PyJWT 1.4.2
    pyOpenSSL 17.4.0
    pyparsing 2.1.10
    PySocks 1.5.7
    python-dateutil 2.5.3
    python-magic 0.4.15
    pytz 2016.10
    PyYAML 3.12
    recordclass 0.4.1
    requests 2.18.4
    requests-oauthlib 0.8.0
    rsa 3.4.2
    s3cmd 2.0.2
    s3transfer 0.1.10
    SecretStorage 2.3.1
    sh 1.12.13
    simplejson 3.16.0
    six 1.11.0
    typing 3.5.3.0
    uritemplate 3.0.0
    urllib3 1.21.1
    responses 0.3.0
    diskcache 3.0.6
    IPy 1.0
Name Author Project Info
python27 -- TOOLS Virtual environment with python 2.7
    Package Version
    awscli 1.16.60
    alembic 0.8.6
    amqp 1.4.9
    anyjson 0.3.3
    Babel 1.3
    backports.ssl-match-hostname 3.5.0.1
    billiard 3.3.0.23
    boto 2.40.0
    bleach 2.0.0
    celery 3.1.23
    certifi 2016.2.28
    chartkick 0.4.2
    croniter 0.3.12
    dill 0.2.5
    Flask 0.10.1
    Flask-Admin 1.4.0
    Flask-Cache 0.13.1
    Flask-Login 0.2.11
    Flask-WTF 0.14
    flower 0.9.2
    future 0.15.2
    futures 3.0.5
    gunicorn 19.3.0
    inflection 0.3.1
    itsdangerous 0.24
    Jinja2 2.8
    kombu 3.0.35
    Mako 1.0.4
    Markdown 2.6.6
    MarkupSafe 0.23
    numpy 1.11.1rc1
    pandas 0.18.1
    prometheus_client 0.4.2
    Pygments 2.1.3
    psycopg2 2.7.1
    python-dateutil 2.5.3
    python-editor 1.0.1
    pytz 2016.4
    qds-sdk 1.10.0
    requests 2.10.0
    setproctitle 1.1.10
    six 1.10.0
    SQLAlchemy 1.1.0b1
    thrift 0.9.3
    tornado 4.2
    urllib3 1.16
    Werkzeug 0.11.10
    wheel 0.24.0
    WTForms 2.1
    lxml 2.3
    bs4 0.0.1
    awscli 1.16.224
    s3cmd 2.0.2
    virtualenv 16.7.3

Qubole Packages


Name Engine Status Package Version S3 link
hustler - Active 0.605.1926
hive13 - Deprecated 0.41.188
pig 0.11 Deprecating Soon 0.72508840.4
pig 0.15 Deprecating Soon 0.46814473.6
pig 0.17 Deprecating Soon 0.14018818.6
hadoop2 2.6.0 Active 0.86036188.360
hadoop2 2.8.1 Active 0.97413592.49
hadoop2 3.1.0 Active 0.13114335.163
hive2 - Active 0.66.213
hive1_2 1.2 Active 0.17390818.99
hive1_2 1.2.1 Active 0.91500599.27
hive1_2 2.1.1 Active 0.47723873.281
hive1_2 2.3 Active 0.72321593.82
hive1_2 3.1.1 Active 0.3705563.99
hbase - Active 0.13206.41
presto 0.157 Active 0.92472999.130
presto 0.180 Active 0.13837551.230
presto 0.193 Active 0.849917.197
presto 0.208 Active 0.80091709.195
spark_common - Active 0.17390818.15
spark 1.5.1 Active 0.26419471.18
spark 1.6.0 Active 0.51764804.74
spark 1.6.1 Active 0.29775358.85
spark 1.6.2 Active 0.84062888.84
spark 2.0.0 Active 0.7541317.139
spark 2.0.2 Active 0.63491637.119
spark 2.1.0 Active 0.53726898.156
spark 2.1.1 Active 0.26272012.125
spark 2.2.0 Active 0.59877903.124
spark 2.2.1 Active 0.54337811.116
spark 2.3.1 Active 0.28257552.70
spark 2.3.2 Active 0.47030478.61
spark 2.4.0 Active 0.14037411.103
spark 2.4.3 Active 0.90944415.40
zeppelin 1.5 Active 0.94066127.326
zeppelin 1.6 Active 0.94066127.311
zeppelin 2.0 Active 0.94066127.299
zeppelin 2.1 Active 0.94066127.290
zeppelin 2.2 Active 0.94066127.222
zeppelin 2.3 Active 0.94066127.118
zeppelin 2.4 Active 0.94066127.86
jupyterlab - Active 0.17390818.69
tez 0.7.0 Active 0.77714688.48
tez 0.8.4 Active 0.41464392.60
tez 0.9.1 Active 0.88541355.21
sqoop_h2 1.4.6 Active 0.53007081.7
sqoop_h2 1.4.7 Active 0.22307209.4
airflow 1.10.0 Active 0.83869506.31
airflow 1.10.2.QDS Active 0.37729048.19
airflow 1.7.0 Active 0.72738640.18
airflow 1.8.2 Active 0.34744164.43
metricsd - Active 0.17390818.815
dr_elephant - Active 0.92021444.704
jeeves - Active 0.17390818.105
client_wrappers - Active 0.17390818.174
quboled - Active 0.17390818.49
knox - Active 0.17390818.23
prometheus - Active 0.17390818.27
cloudman - Active 0.140.484
megamind - Active 0.17390818.16
qds_ml - Active 0.17390818.3
ml_metastore - Active 0.17390818.4
logan - Active 0.17390818.10
rstudio_utils fuse_s3fs Active 0.17390818.5
rstudio_utils rsd Active 0.17390818.6
rubix - Active 0.17390818.18
file_merge_utility - Active 0.17390818.7
sparklens - Active 0.17390818.13