accasim.utils.misc module

MIT License

Copyright (c) 2017 cgalleguillosm, AlessioNetti

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

class accasim.utils.misc.CONSTANT

Bases: accasim.utils.misc.Singleton

This class allows to load all config parameters into a Singleton Object. This object will allow access to all the parameters. The parameters could be accessed as attribute name.

New attrs could be passed as dict (load_constants()) or simply with (attr, value) (load_constant())

Example:

Program:

>>> PATH = '/path/to/'
>>> c = CONSTANT()
>>> c.load('PATH', PATH)
>>> print(c.PATH)
>>> /path/to/

Note:

It’s loaded into all base class by default!

clean_constants()

load_constant(k, v)

Load an specific parameter.

Parameters:

• k – Name of the parameter
• v – Value of the parameter

load_constants(_dict)

Loads an entire dictionary into the singleton.

Parameters:_dict – Dictionary with the new parameters to load.

class accasim.utils.misc.FrozenDict(*args, **kwargs)

Bases: collections.abc.Mapping

Inmutable dictionary useful for storing parameters that are dinamycally loaded

class accasim.utils.misc.Singleton

Bases: object

Singleton class

class accasim.utils.misc.SystemStatus(port, functions)

Bases: object

Wathcer Daemon allows to track the simulation process through command line querying.

MAX_LENGTH = 2048

call_inner_function(name)

Call a function and retrives it results

Parameters:name – name of the function

listenForRequests()

Listening for requests

start()

Start the daemon

stop()

Stop the daemon

accasim.utils.misc.clean_results(*args)

Removes the filepaths passed as argument

Parameters:*args – List of filepaths

accasim.utils.misc.cmp_to_key(mycmp)

Convert a cmp= function into a key= function

accasim.utils.misc.default_sorted_attribute(workload_line, attr='submit_time', converter=None)

Parameters:

• workload_line – A line readed from the file.
• attr – Attribute of the line for sorting.
• converter – Converter function to cast the attribute.

Returns:

Returns the attribute of the line. Casted if it’s required.

accasim.utils.misc.default_sorting_function(obj1, obj2, avoid_data_tokens=[';'])

Function for sorting the swf files in ascending order. If one of the object belongs to avoid_data_tokens, the same order is maintained by returning 1.

Parameters:

• obj1 – Object 1
• obj2 – Object 2
• avoid_data_tokens – Tokens to avoid

Returns:

return a positive number for maintaing the order, or a negative one to change the order.

accasim.utils.misc.from_isodatetime_2_timestamp(dtime, hours=0, minutes=0)

Converts a ISO datetime to Unix Timestamp

Parameters:

• dtime – Datetime in YYYY-MM-DD HH:MM:SS format
• hours – Hours to adjust the timezone
• minutes – Minutes to adjust the timezone

Returns:

Timestamp of the dtime

accasim.utils.misc.generate_config(config_fp, **kwargs)

Creates a config file.

Parameters:

• config_fp – Filepath to the config
• **kwargs – Source for the config data

accasim.utils.misc.hinted_tuple_hook(obj)

Decoder for specific object of json files, for preserving the type of the object. It’s used with the json.load function.

accasim.utils.misc.list_class_assertion(_list, class_type, allow_empty=False, error_msg='List class error exception.{}', msg_args=None)

Parameters:

• _list –
• class_type –
• allow_empty –
• error_msg –
• msg_args –

Returns:

accasim.utils.misc.load_config(config_fp)

Loads an specific config file in json format

Parameters:config_fp – Filepath of the config file. Returns:Dictionary with the configuration.

accasim.utils.misc.obj_assertion(obj, class_type, error_msg=None, msg_args=None)

Parameters:

• obj –
• class_type –
• error_msg –
• msg_args –

Returns:

accasim.utils.misc.sort_file(input_filepath, lines=None, sort_function=<function default_sorting_function>, avoid_data_tokens=[';'], output_filepath=None)

The input file for the simulator must be sorted by submit time. It modifies the file input file, or also can be saved to a new one if the output_filepath arg is defined.

Parameters:

• input_filepath – Input workload file
• lines – Number of lines to be read. It includes all lines from the begining of the file.
• sort_function – (Optional) The function that sorts the file by submit time. The user is responsable to define the correct function. If a workload with SWF format is used, by default default_sorting_function (SWF workload) is used.
• avoid_data_tokens – (Optional) By default avoid to modify comment lines of SWF workload.
• output_filepath – (Optional) The sorted data is saves into another file (this filepath). It will not content the lines that begin with tokens of the avoid_data_tokens var.

Returns:

A list of queued time points.

class accasim.utils.misc.str_(text)

Bases: object

class accasim.utils.misc.str_datetime(epoch_time)

Bases: object

REGEX = '\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}'

REGEX_GROUP = '(?P<{}>\\d{{4}}-\\d{{2}}-\\d{{2}}\\s\\d{{2}}:\\d{{2}}:\\d{{2}})'

get_hours()

get_minutes()

get_month()

From 1 to 12

get_weekday()

From 1 to 7

class accasim.utils.misc.str_nodes(nodes)

Bases: object

class accasim.utils.misc.str_resources(nodes, resources)

Bases: object

REGEX = '[\\d+;|#]+'

REGEX_GROUP = '(?P<{}>[\\d+;|#]+)'

SEPARATOR = '#'

class accasim.utils.misc.str_time(secs)

Bases: object

accasim.utils.misc.type_regexp(_type, new_regexp={})

Parameters:

• _type –
• new_regexp –

Returns:

accasim.utils.misc.workload_parser(workload_line, attrs=None, avoid_data_tokens=[';'])

Attributes of each workload line in a SWF format (separated by space):

1. job_number – a counter field, starting from 1.
2. submit_time – in seconds. The earliest time the log refers to is zero, and is usually the submittal time of the first job. The lines in the log are sorted by ascending submittal times. It makes sense for jobs to also be numbered in this order.
3. wait_time – in seconds. The difference between the job’s submit time and the time at which it actually began to run. Naturally, this is only relevant to real logs, not to models.
4. duration – in seconds. The wall clock time the job was running (end time minus start time).
5. allocated_processors – an integer. In most cases this is also the number of processors the job uses; if the job does not use all of them, we typically don’t know about it.
6. avg_cpu_time – Time Used for both user and system, in seconds. This is the average over all processors of the CPU time used, and may therefore be smaller than the wall clock runtime. If a log contains the total CPU time used by all the processors, it is divided by the number of allocated processors to derive the average.
7. used_memory – in kilobytes. This is again the average per processor.
8. requested_number_processors — Requested Number of Processors.
9. requested_time – This can be either runtime (measured in wallclock seconds), or average CPU time per processor (also in seconds) – the exact meaning is determined by a header comment. In many logs this field is used for the user runtime estimate (or upper bound) used in backfilling. If a log contains a request for total CPU time, it is divided by the number of requested processors.
10. requested_memory – Requested memory in kilobytes per processor.
11. status – 1 if the job was completed, 0 if it failed, and 5 if cancelled. If information about chekcpointing or swapping is included, other values are also possible. See usage note below. This field is meaningless for models, so would be -1.
12. user_id – a natural number, between one and the number of different users.
13. group_id – a natural number, between one and the number of different groups. Some systems control resource usage by groups rather than by individual users.
14. executable_number – a natural number, between one and the number of different applications appearing in the workload. in some logs, this might represent a script file used to run jobs rather than the executable directly; this should be noted in a header comment.
15. queue_number – a natural number, between one and the number of different queues in the system. The nature of the system’s queues should be explained in a header comment. This field is where batch and interactive jobs should be differentiated: we suggest the convention of denoting interactive jobs by 0.
16. partition_number – a natural number, between one and the number of different partitions in the systems. The nature of the system’s partitions should be explained in a header comment. For example, it is possible to use partition numbers to identify which machine in a cluster was used.
17. preceding_job_number – this is the number of a previous job in the workload, such that the current job can only start after the termination of this preceding job. Together with the next field, this allows the workload to include feedback as described below.
18. think_time_prejob – this is the number of seconds that should elapse between the termination of the preceding job and the submittal of this one.

Parameters:

• workload_line – A Line of the workload file
• attrs – List of attributes to be considered. Default None, all attributes will be considered.
• avoid_data_tokens – List of tokens to avoid the line

Returns:

A dictionary with all the attributes requested. If the line is returned it means that the line has the token to avoid.