Logging (core)#

The framework allows to log sent messages for each GUILD/USER (if you set the “logging” to True inside the GUILD or USER object).

Logging is handled thru so called logging managers. Currently, 3 different managers exists:

If a logging managers fails saving a log, then it’s fallback manager will be used temporarily to store the log. It will only use the fallback once and then, at the next message, the original manager will be used.

../../_images/logging_process.drawio.svg

Fig. 4 Logging process with fallback#

JSON Logging (file)#

The logs are written in the JSON format and saved into a JSON file, that has the name of the guild or an user you were sending messages into. The JSON files are fragmented by day and stored into folder Year/Month/Day, this means that each day a new JSON file will be generated for that specific day for easier managing, for example, if today is 13.07.2022, the log will be saved into the file that is located in

History
└───2022
│   └───07
│       └───13
|           └─── #David's dungeon.json

JSON structure#

The log structure is the same for both USER and GUILD. All logs will contain keys:

JSON code example#

Listing 10 Code to produce JSON logs#
from datetime import timedelta
import daf

rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]

@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item

accounts = [
    daf.ACCOUNT( token="SDASKDKLSADJKLSDJ",
                 is_user=False,
                 servers=[ 
                    daf.GUILD(12345, [daf.TextMESSAGE(None, timedelta(seconds=5), get(rolls.copy()), [12345], "edit")], True) 
                  ] )
]


daf.run(
    accounts=accounts,
    logger=daf.LoggerJSON("History")
)

CSV Logging (file)#

The logs are written in the CSV format and saved into a CSV file, that has the name of the guild or an user you were sending messages into. The CSV files are fragmented by day and stored into folder Year/Month/Day, this means that each day a new CSV file will be generated for that specific day for easier managing, for example, if today is 13.07.2023, the log will be saved into the file that is located in

History
└───2023
│   └───07
│       └───13
|           └─── #David's dungeon.csv

CSV structure#

The structure contains the following attributes:

  • Timestamp (string)

  • Guild Type (string),

  • Guild Name (string),

  • Guild Snowflake (integer),

  • Message Type (string),

  • Sent Data (json),

  • Message Mode (non-empty for TextMESSAGE and DirectMESSAGE) (string),

  • Message Channels (non-empty for TextMESSAGE and VoiceMESSAGE) (json),

  • Success Info (non-empty for DirectMESSAGE) (json),

Note

Attributes marked with (json) are the same as in JSON Logging (file)

CSV code example#

Listing 11 Code to produce JSON logs#
from datetime import timedelta
import daf

rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]

@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item

accounts = [
    daf.ACCOUNT( token="SDASKDKLSADJKLSDJ",
                 is_user=False,
                 servers=[ 
                    daf.GUILD(12345, [daf.TextMESSAGE(None, timedelta(seconds=5), get(rolls.copy()), [12345], "edit")], True) 
                  ] )
]


daf.run(
    accounts=accounts,
    logger=daf.LoggerCSV("History", ";")
)

Relational Database Log (SQL)#

Changed in version v2.6: Added author parameter to MessageLOG

This type of logging enables saving logs to a remote server inside the database. In addition to being smaller in size, database logging takes up less space and it allows easier data analysis.

../../_images/sql_logging_process.drawio.svg

Fig. 5 SQL Logging diagram#

Dialects#

The dialect is selected via the dialect parameter in LoggerSQL. The following dialects are supported:

  • Microsoft SQL Server

  • PostgreSQL

  • SQLite,

  • MySQL

Usage#

For daf to use SQL logging, you need to pass the run() function with the logger parameter and pass it the LoggerSQL object.

from datetime import timedelta
import daf

rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]

@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item

accounts = [
    daf.ACCOUNT( token="SDASKDKLSADJKLSDJ",
                 is_user=False,
                 servers=[ 
                    daf.GUILD(12345, [daf.TextMESSAGE(None, timedelta(seconds=5), get(rolls.copy()), [12345], "edit")], True) 
                  ] )
]



daf.run(
    #logger=daf.LoggerSQL(dialect="mysql", username="user", password="pass", database="TestDB", server="127.0.0.1"),
    logger=daf.LoggerSQL(dialect="postgresql", username="postgres", password="pass", database="TestDB", server="127.0.0.1"),
    #logger=daf.LoggerSQL(dialect="mssql", username="sa", password="pass", database="TestDB", server="127.0.0.1"),
    #logger=daf.LoggerSQL(dialect="sqlite", database="testdb"),
    accounts=accounts
)

Features#

  • Automatic creation of the schema

  • Caching for faster logging

  • Low redundancy for reduced file size

  • Automatic error recovery

Note

The database must already exist! However it can be completely empty, no need to manually create the schema.

ER diagram#

../../_images/sql_er.drawio.svg

Analysis#

The LoggerSQL provides some methods for data analysis:

Tables#

MessageLOG#

Description:

This table contains the actual logs of sent messages, if the message type is DirectMESSAGE, then all the information is stored in this table. If the types are Voice/Text MESSAGE, then channel part of the log is saved in the MessageChannelLOG table.

Attributes:
  • [Primary Key] id: Integer - This is an internal ID of the log inside the database.

  • sent_data: Integer - Foreign key pointing to a row inside the DataHISTORY table.

  • message_type: SmallInteger - Foreign key ID pointing to a entry inside the MessageTYPE table.

  • guild_id: Integer - Foreign key pointing to GuildUSER table, represents guild id of guild the message was sent into.

  • author_id: Integer - Foreign key pointing to GuildUSER table, represents the author account of the message.

  • message_mode: SmallInteger - Foreign key pointing to MessageMODE table. This is non-null only for DirectMESSAGE.

  • dm_reason: String - If MessageTYPE is not DirectMESSAGE or the send attempt was successful, this is NULL, otherwise it contains the string representation of the error that caused the message send attempt to be unsuccessful.

  • timestamp: DateTime - The timestamp of the message send attempt.

DataHISTORY#

Description:

This table contains all the different data that was ever advertised. Every element is unique and is not replicated. This table exist to reduce redundancy and file size of the logs whenever same data is advertised multiple times. When a log is created, it is first checked if the data sent was already sent before, if it was the id to the existing DataHISTORY row is used, else a new row is created.

Attributes:
  • [Primary Key] id: Integer - Internal ID of data inside the database.

  • content: JSON - Actual data that was sent.

MessageTYPE#

Description:

This is a lookup table containing the the different message types that exist within the framework (Messages).

Attributes:
  • [Primary Key] id: SmallInteger - Internal ID of the message type inside the database.

  • name: String - The name of the actual message type.

GuildUSER#

Description:

The table contains all the guilds/users the framework ever generated a log for and all the authors.

Attributes:
  • [Primary Key] id: Integer - Internal ID of the Guild/User inside the database.

  • snowflake_id: BigInteger - The discord (snowflake) ID of the User/Guild

  • name: String - Name of the Guild/User

  • guild_type: SmallInteger - Foreign key pointing to GuildTYPE table.

MessageMODE#

Description:

This is a lookup table containing the the different message modes available by TextMESSAGE / DirectMESSAGE, it is set to null for VoiceMESSAGE.

Attributes:
  • [Primary Key] id: SmallInteger - Internal identifier of the message mode inside the database.

  • name: String - The name of the actual message mode.

GuildTYPE#

Description:

This is a lookup table containing types of the guilds inside the framework (Guilds).

Attributes:
  • [Primary Key] id: SmallInteger - Internal identifier of the guild type inside the database.

  • name: String - The name of the guild type.

CHANNEL#

Description:

The table contains all the channels that the framework ever advertised into.

Attributes:
  • [Primary Key] id: Integer - Internal identifier of the channel inside the database

  • snowflake_id: BigInteger - The discord (snowflake) identifier representing specific channel

  • name: String - The name of the channel

  • guild_id: Integer - Foreign key pointing to a row inside the GuildUSER table. It points to a guild that the channel is part of.

MessageChannelLOG#

Description:

Since messages can send into multiple channels, each MessageLOG has multiple channels which cannot be stored inside the MessageLOG. This is why this table exists. It contains channels of each MessageLOG.

Attributes:
  • [Primary Key] [Foreign Key] log_id: Integer - Foreign key pointing to a row inside MessageLOG (to which log this channel log belongs to).

  • [Primary Key] [Foreign Key] channel_id: Integer - Foreign key pointing to a row inside the CHANNEL table.

  • reason: String - Reason why the send failed or NULL if send succeeded.

Custom Logger#

If you want to use a different logging scheme than the ones built in, you can do so by creating a custom logging manager that inherits the daf.logging.LoggerBASE.

The derived logger class can then implement the following methods:

  1. __init__(self, param1, param2, …) [Required]:

    The method used for passing parameters and for basic non-async initialization. This method must contain a fallback parameter and also needs to have an attribute of the same name.

    Listing 12 Custom __init__ method#
     class LoggerCUSTOM(daf.logging.LoggerBASE):
        def __init__(self, ..., logger):
            ... # Set attributes
            super().__init__(logger)
    
        ... # Other methods
    
  2. async initialize(self) [Optional]:

    The base’s initialize method calls initialize method of it’s fallback, if it fails then the fallback is set to None.

    If you wish to do additional initialization that requires async/await operations, you can implement your own initialize method but make sure you call the base’s method in the end.

    Listing 13 Custom initialize method#
    class LoggerCUSTOM(daf.logging.LoggerBASE):
        ... # Other methods
    
        async def initialize(self):
            ... # Custom implementation code
            await super().initialize()
    
  3. async _save_log(self, guild_context: dict, message_context: dict) [Required]:

    Method that stores the message log. If there is any error in saving the log an exception should be raised, which will then make the logging module automatically use the fallback manager, do not call the fallback manager from this method!

    Parameters:

    guild_context (dict) - Contains keys:

    • “name”: The name of the guild/user (str)

    • “id”: Snowflake ID of the guild/user (int)

    • “type”: object type (GUILD/USER) that generated the log. (str)

    message_context (dict) - Dictionary returned by:

  4. async update(self, **kwargs) [Optional]:

    Custom implementation of the update method.

    This method is used for updating the parameters that are available thru __init__ method and is not required if the attributes inside the object have the same name as the parameters inside the __init__ function and there are no pre-required steps that need to be taken before updating (see JSON Logging (file)’s code for example).

    However if the name of attributes differ from parameter name or the attribute doesn’t exist at all or other steps are required than just re-initialization (see daf.logging.sql.LoggerSQL’s update method), then this method is required to be implemented. It should be implemented in a way that it calls the base update method. Example:

    class LoggerCUSTOM(daf.logging.LoggerBASE):
        def __init__(self, name, fallback):
            self._name = name
            super().__init__(fallback)
    
        ... # Other methods
    
        async def update(self, **kwargs)
            # Only modify if the parameter is not passed to update method
            if "name" not kwargs:
                # The name parameter is stored under "_name" attribute instead of "name"
                kwargs["name"] = self._name
    
            ... # Other pre-required code (eg. remote SQL server needs to be disconnected)
    
            super().update(**kwargs) # Call base update method