java - How do i get started with Amazon Web Services for this scenario? -

java - How do i get started with Amazon Web Services for this scenario? -

i'm used having remote server can utilize via ssh looking @ using amazon web services new project give me improve performance , resilience @ reduced costs i'm struggling understand how utilize it.

this want do:

first-time:

create postgres db connect amazon server download code server

once month:

download big info file server run code (written in python) load database data run code (written in java) create lucene search index file info in database

continuously:

run java code in servlet container, utilize lucense search index file not require access database.

note: technically create database population locally problem resultant lucene index file 5gb , dont have plenty net connection upload file of size amazon.

all have managed far create postgres database don't understand how connect or ssh/telnet connection server (i requested direct connect seems different service).

update far

fyi:

so created postgres database using rds i created ubuntu linux installation using ec2 i connected linux installation using ssh i installed required software (using apt-get) i downloaded info file linux installation

i think according installation should able connect postgres db ec2 instance , local machine however in both cases times out.

* update 2 **

probably security related cannot life of me understand i'm meant security groups ands why don't create ec2 instance able talk database default.

ive checked both rds , ec2 have the3 same vpc id, , both in same availability zone. postgres using port 5432 (not 3306) havent been able access yet. taking working ec2 instance starting point should create new security grouping before creating database, , if values need set can access db psql within ec2 ssh session - thats holding me , need

* update 3 *

at lastly have access database, database had 3 security groups ( think other 2 created when created new ec2 instance) removed 2 of them , in remaining on inbound tab set rule to

all traffic ports 0-65535 protocol all ipaddress 0.0.0.0/0

(the outbound tab had same rule) , worked !

i realize not secure setup @ to the lowest degree progress.

i assume allow access ec2 instance can alter ipaddress of inbound rule don't how calculate cidr ipaddress ?

my new problem having downloaded datafile ec2 instance unable unzip because don't not have plenty diskspace. assume have utilize s3 ive created bucket how create visible diskspace ec2 instance can

move datafile it unzip datafile it run code against unzipped datafile load database

(note datafile xml format , has processed custom code database cannot loaded straight database using generic tool)

update 4

s3 wrong solution me instead can utilize ebs disc storage accessible not service clicking volumes in ec2 console. ensure create volume in same availability zone instance, there maybe more 1 in each location, illustration ec2 instance created in eu-west-1a first time created volume in eu-west-1b , hence not used.

then attach volume instance

but cannot see volume linux commandline, seems there else required.

update 5

okay, have format disk , mount in linux work have code uploading info database working running incredibly slow, much slower inexpensive local server have @ home. i'm guessing because info beingness loaded 1 record @ time bottleneck not micro database micro instance, looks need redo more expensive instance

update 6

updated big computative instance still slow. im thinking issue network latency between server , database perhaps need install postgres server straight onto instance cutting part out.

first-time:

create postgres db - depending on size(small or large), might want rds or redshift connect amazon server - ec2 download code server - upload programs s3 bucket 1 time month:

download big info file server - move info s3, if using redshift info can loaded straight s3 redshift

run code (written in python) load database info run code (written in java) create lucene search index file info in database - might want emr continuously:

run java code in servlet container, utilize lucense search index file not require access database - if have java war file, can host using elasticbean stalk

in order connect database, must create sure security grouping allows connection, , ec2 must create sure port 22 open ip conncet it. sounds security grouping rds isn't opening port 3306.

java python amazon-web-services amazon-ec2