|
Server : Apache System : Linux iad1-shared-b8-43 6.6.49-grsec-jammy+ #10 SMP Thu Sep 12 23:23:08 UTC 2024 x86_64 User : dh_edsupp ( 6597262) PHP Version : 8.2.26 Disable Function : NONE Directory : /lib/python3/dist-packages/boto/emr/__pycache__/ |
Upload File : |
o
ckF[3# � @ s� d dl mZ G dd� de�ZG dd� de�ZG dd� de�ZG dd � d e�ZG d
d � d e�ZG d d
� d
e�ZG dd� de�Z G dd� de�Z
G dd� de
�Z G dd� de
�Z dS )� )�sixc @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) �Stepz!
Jobflow Step base class
c C � t � �)z=
:rtype: str
:return: URI to the jar
��NotImplemented��self� r �//usr/lib/python3/dist-packages/boto/emr/step.py�jar � zStep.jarc C r )zS
:rtype: list(str)
:return: List of arguments for the step
r r r r r
�args% r z Step.argsc C r )zB
:rtype: str
:return: The main class name
r r r r r
�
main_class, r zStep.main_classN)�__name__�
__module__� __qualname__�__doc__r r
r r r r r
r s
r c @ s6 e Zd ZdZ d dd�Zdd� Zdd � Zd
d � ZdS )
�JarStepz
Custom jar step
N�TERMINATE_JOB_FLOWc C s4 || _ || _|| _|| _t|tj�r|g}|| _dS )a
A elastic mapreduce step that executes a jar
:type name: str
:param name: The name of the step
:type jar: str
:param jar: S3 URI to the Jar file
:type main_class: str
:param main_class: The class to execute in the jar
:type action_on_failure: str
:param action_on_failure: An action, defined in the EMR docs to
take on failure.
:type step_args: list(str)
:param step_args: A list of arguments to pass to the step
N)�name�_jar� _main_class�action_on_failure�
isinstancer � string_types� step_args)r r r r r r r r r
�__init__8 s
zJarStep.__init__c C � | j S �N�r r r r r
r S � z JarStep.jarc C s g }| j r |�| j � |S r )r �extend)r r
r r r
r
V s z JarStep.argsc C r r )r r r r r
r ^ r zJarStep.main_class)Nr N)r r r r r r r
r r r r r
r 4 s
� r c @ sD e Zd ZdZ ddd�Zdd� Zd d
� Zd d � Zd
d� ZdS )�
StreamingStepz
Hadoop streaming step
Nr �3/home/hadoop/contrib/streaming/hadoop-streaming.jarc C sX || _ || _|| _|| _|| _|| _|| _| | _|
| _| | _ t
|t j �r'|g}|| _
dS )a�
A hadoop streaming elastic mapreduce step
:type name: str
:param name: The name of the step
:type mapper: str
:param mapper: The mapper URI
:type reducer: str
:param reducer: The reducer URI
:type combiner: str
:param combiner: The combiner URI. Only works for Hadoop 0.20
and later!
:type action_on_failure: str
:param action_on_failure: An action, defined in the EMR docs to
take on failure.
:type cache_files: list(str)
:param cache_files: A list of cache files to be bundled with the job
:type cache_archives: list(str)
:param cache_archives: A list of jar archives to be bundled with
the job
:type step_args: list(str)
:param step_args: A list of arguments to pass to the step
:type input: str or a list of str
:param input: The input uri
:type output: str
:param output: The output uri
:type jar: str
:param jar: The hadoop streaming jar. This can be either a local
path on the master node, or an s3:// URI.
N)r �mapper�reducer�combinerr � cache_files�cache_archives�input�outputr r r r r ) r r r$ r% r& r r' r( r r) r* r r r r
r f s #
zStreamingStep.__init__c C r r r r r r r
r � r zStreamingStep.jarc C s d S r r r r r r
r � s zStreamingStep.main_classc C s� g }| j r |�| j � |�d| jg� | jr |�d| jg� | jr*|�d| jg� n|�ddg� | jrPt| jt�rH| jD ] }|�d|f� q=n|�d| jf� | jr[|�d| jf� | j rk| j D ] }|�d|f� qa| j
r{| j
D ] }|�d |f� qq|S )
Nz-mapperz -combinerz-reducerz-jobconfzmapred.reduce.tasks=0z-inputz-outputz
-cacheFilez
-cacheArchive) r r! r$ r& r% r) r �listr* r' r( )r r
r) �
cache_file�
cache_archiver r r
r
� s0
�
zStreamingStep.argsc
C s< d| j j| j j| j| j| j| j| j| j| j | j
| j | j f S )Nz�%s.%s(name=%r, mapper=%r, reducer=%r, action_on_failure=%r, cache_files=%r, cache_archives=%r, step_args=%r, input=%r, output=%r, jar=%r))
� __class__r r r r$ r% r r' r( r r) r* r r r r r
�__repr__� s �zStreamingStep.__repr__) NNr NNNNNr# ) r r r r r r r r
r/ r r r r
r" b s
�3 %r" c s e Zd ZdZ� fdd�Z� ZS )�ScriptRunnerStepzEs3n://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jarc s t t| �j|| jfi |�� d S r )�superr0 r �ScriptRunnerJar)r r �kw�r. r r
r � s zScriptRunnerStep.__init__)r r r r2 r �
__classcell__r r r4 r
r0 � s r0 c @ � e Zd Zg d�ZdS )�PigBase)z4s3n://us-east-1.elasticmapreduce/libs/pig/pig-script� --base-pathz*s3n://us-east-1.elasticmapreduce/libs/pig/N�r r r �BaseArgsr r r r
r7 � � r7 c s& e Zd ZdZdZd� fdd� Z� ZS )�InstallPigStepz!
Install pig on emr step
z Install Pig�latestc sD g }|� | j� |� dg� |� d|g� tt| �j| j|d� d S )Nz
--install-pig�--pig-versions�r )r! r: r1 r< r �InstallPigName)r � pig_versionsr r4 r r
r � s
zInstallPigStep.__init__)r= )r r r r r@ r r5 r r r4 r
r< � s r< c s& e Zd ZdZdg f� fdd� Z� ZS )�PigStepz
Pig script step
r= c sR g }|� | j� |� d|g� |� ddd|g� |� |� tt| �j||d� d S )Nr> z--run-pig-script�--args�-fr? )r! r: r1 rB r )r r �pig_filerA �pig_argsr r4 r r
r � s
zPigStep.__init__�r r r r r r5 r r r4 r
rB � s rB c @ r6 )�HiveBase)z6s3n://us-east-1.elasticmapreduce/libs/hive/hive-scriptr8 z+s3n://us-east-1.elasticmapreduce/libs/hive/Nr9 r r r r
rH � r; rH c s& e Zd ZdZdZd� fdd� Z� ZS )�InstallHiveStepz"
Install Hive on EMR step
z Install Hiver= Nc s\ g }|� | j� |� dg� |� d|g� |d ur!|� d| g� tt| �j| j|d� d S )Nz--install-hive�--hive-versionsz--hive-site=%sr? )r! r: r1 rI r �InstallHiveName)r �
hive_versions� hive_siter r4 r r
r s
�zInstallHiveStep.__init__�r= N)r r r r rK r r5 r r r4 r
rI � s rI c s& e Zd ZdZ d� fdd� Z� ZS )�HiveStepz
Hive script step
r= Nc sZ g }|� | j� |� d|g� |� ddd|g� |d ur!|� |� tt| �j||d� d S )NrJ z--run-hive-scriptrC rD r? )r! r: r1 rO r )r r � hive_filerL � hive_argsr r4 r r
r s
zHiveStep.__init__rN rG r r r4 r
rO s
�rO N)
� boto.compatr �objectr r r" r0 r7 r< rB rH rI rO r r r r
�<module> s .j