Logging#

The framework allows to log sent messages for each GUILD/USER (if you set the “logging” to True inside the GUILD or USER object).

Logging is handled thru so called logging managers. Currently, 3 different managers exists:

LoggerJSON: Used for saving file logs in the JSON format. (JSON Logging (file))
LoggerCSV: Used for saving file logs in the CSV format, where certain fields are still JSON. (CSV Logging (file))
LoggerSQL: Used for saving relational database logs into a remote database. (Relational Database Log (SQL))
Custom logger: User can create a custom logger if they desire. (Custom Logger)

If a logging managers fails saving a log, then it’s fallback manager will be used temporarily to store the log. It will only use the fallback once and then, at the next message, the original manager will be used.

../_images/logging_process.png — Fig. 1 Logging process with fallback#

JSON Logging (file)#

The logs are written in the JSON format and saved into a JSON file, that has the name of the guild or an user you were sending messages into. The JSON files are fragmented by day and stored into folder Year/Month/Day, this means that each day a new JSON file will be generated for that specific day for easier managing, for example, if today is 13.07.2022, the log will be saved into the file that is located in

History
└───2022
│   └───07
│       └───13
|           └─── #David's dungeon.json

JSON structure#

The log structure is the same for both USER and GUILD. All logs will contain keys:

“name”: The name of the guild/user
“id”: Snowflake ID of the guild/user
“type”: object type (GUILD/USER) that generated the log.
“message_history”: Array of logs for each sent message to the guild/user, the structure is message type dependant and is generated inside methods:

JSON code example#

Listing 8 Code to produce JSON logs#

from datetime import timedelta
import daf

rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]

@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item

servers = [
    daf.GUILD(
        snowflake=12345,
        messages=[
            daf.TextMESSAGE(None, timedelta(seconds=5), get(rolls.copy()), [12345], "edit", timedelta(seconds=5))
        ],
        logging=True
    )
]


daf.run(
    token="OSDSJ44JNnnJNJ2NJDBWQUGHSHFAJSHDUQHFDBADVAHJVERAHGDVAHJSVDE",   # Example account token
    is_user=False, 
    server_list=servers,
    logger=daf.LoggerJSON("History")
)

CSV Logging (file)#

The logs are written in the CSV format and saved into a CSV file, that has the name of the guild or an user you were sending messages into. The CSV files are fragmented by day and stored into folder Year/Month/Day, this means that each day a new CSV file will be generated for that specific day for easier managing, for example, if today is 13.07.2023, the log will be saved into the file that is located in

History
└───2023
│   └───07
│       └───13
|           └─── #David's dungeon.csv

CSV structure#

The structure contains the following attributes:

Timestamp (string)
Guild Type (string),
Guild Name (string),
Guild Snowflake (integer),
Message Type (string),
Sent Data (json),
Message Mode (non-empty for TextMESSAGE and DirectMESSAGE) (string),
Message Channels (non-empty for TextMESSAGE and VoiceMESSAGE) (json),
Success Info (non-empty for DirectMESSAGE) (json),

Note

Attributes marked with (json) are the same as in JSON Logging (file)

CSV code example#

Listing 9 Code to produce JSON logs#

from datetime import timedelta
import daf

rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]

@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item

servers = [
    daf.GUILD(
        snowflake=12345,
        messages=[
            daf.TextMESSAGE(None, timedelta(seconds=5), get(rolls.copy()), [12345], "edit", timedelta(seconds=5))
        ],
        logging=True
    )
]


daf.run(
    token="OSDSJ44JNnnJNJ2NJDBWQUGHSHFAJSHDUQHFDBADVAHJVERAHGDVAHJSVDE",   # Example account token
    is_user=False, 
    server_list=servers,
    logger=daf.LoggerCSV("History", ";")
)

Relational Database Log (SQL)#

New in version v1.9.

Changed in version v2.1:

Turned into an optional feature.

pip install discord-advert-framework[sql]

Changed in version v2.2:

Additional dialect support:: Microsoft SQL Server, PostgreSQL, MariaDB/MySQL, SQLite
Better Caching:: Improved caching to significantly increase logging speed
asynchronous:: All of the SQL connectors except MS SQL Server are asynchronous.

This type of logging enables saving logs to a remote server inside the database. In addition to being smaller in size, database logging takes up less space and it allows easier data analysis.

Dialects#

The dialect is selected via the dialect parameter in LoggerSQL. The following dialects are supported:

Microsoft SQL Server
PostgreSQL
SQLite,
MySQL

SQL process diagram#

SQL initialization

SQL logging process

Usage#

For daf to use SQL logging, you need to pass the run() function with the logging parameter and pass it the LoggerSQL object.

from datetime import timedelta
import daf
rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]

@daf.data_function
def get(st: list):
    item = st.pop(0)
    st.append(item)
    return item


S = [
daf.GUILD(
            snowflake=123456789,
            messages=[
                daf.TextMESSAGE(None, timedelta(seconds=2), get(rolls), [123456789]),
            ],
            logging=True
        )
]



daf.run(
    token="SDAJHDASHDJKAHDSKAHDKASHDAHDASJKHDAKJHDAKSHDKASHDKSJHD",
    #logger=daf.LoggerSQL(dialect="mysql", username="user", password="pass", database="TestDB", server="127.0.0.1"),
    logger=daf.LoggerSQL(dialect="postgresql", username="postgres", password="pass", database="TestDB", server="127.0.0.1"),
    #logger=daf.LoggerSQL(dialect="mssql", username="sa", password="pass", database="TestDB", server="127.0.0.1"),
    #logger=daf.LoggerSQL(dialect="sqlite", database="testdb"),
    server_list=S
)

Features#

Automatic creation of the schema
Caching for faster logging
Low redundancy for reduced file size
Automatic error recovery

Note

The database must already exist! However it can be completely empty, no need to manually create the schema.

ER diagram#

Tables#

MessageLOG#

Description:

This table contains the actual logs of sent messages, if the message type is DirectMESSAGE, then all the information is stored in this table. If the types are Voice/Text MESSAGE, then channel part of the log is saved in the MessageChannelLOG table.

Attributes:

[Primary Key] id: Integer - This is an internal ID of the log inside the database.
sent_data: Integer - Foreign key pointing to a row inside the DataHISTORY table.
message_type: SmallInteger - Foreign key ID pointing to a entry inside the MessageTYPE table.
guild_id: Integer - Foreign key pointing to GuildUSER table.
message_mode: SmallInteger - Foreign key pointing to MessageMODE table. This is non-null only for DirectMESSAGE.
dm_reason: String - If MessageTYPE is not DirectMESSAGE or the send attempt was successful, this is NULL, otherwise it contains the string representation of the error that caused the message send attempt to be unsuccessful.
timestamp: DateTime - The timestamp of the message send attempt.

DataHISTORY#

Description:

This table contains all the different data that was ever advertised. Every element is unique and is not replicated. This table exist to reduce redundancy and file size of the logs whenever same data is advertised multiple times. When a log is created, it is first checked if the data sent was already sent before, if it was the id to the existing DataHISTORY row is used, else a new row is created.

Attributes:

[Primary Key] id: Integer - Internal ID of data inside the database.
content: JSON - Actual data that was sent.

MessageTYPE#

Description:

This is a lookup table containing the the different message types that exist within the framework (Messages).

Attributes:

[Primary Key] id: SmallInteger - Internal ID of the message type inside the database.
name: String - The name of the actual message type.

GuildUSER#

Description:

The table contains all the guilds/users the framework ever generated a log for.

Attributes:

[Primary Key] id: Integer - Internal ID of the Guild/User inside the database.
snowflake_id: BigInteger - The discord (snowflake) ID of the User/Guild
name: String - Name of the Guild/User
guild_type: SmallInteger - Foreign key pointing to GuildTYPE table.

MessageMODE#

Description:

This is a lookup table containing the the different message modes available by TextMESSAGE / DirectMESSAGE, it is set to null for VoiceMESSAGE.

Attributes:

[Primary Key] id: SmallInteger - Internal identifier of the message mode inside the database.
name: String - The name of the actual message mode.

GuildTYPE#

Description:

This is a lookup table containing types of the guilds inside the framework (Guilds).

Attributes:

[Primary Key] id: SmallInteger - Internal identifier of the guild type inside the database.
name: String - The name of the guild type.

CHANNEL#

Description:

The table contains all the channels that the framework ever advertised into.

Attributes:

[Primary Key] id: Integer - Internal identifier of the channel inside the database
snowflake_id: BigInteger - The discord (snowflake) identifier representing specific channel
name: String - The name of the channel
guild_id: Integer - Foreign key pointing to a row inside the GuildUSER table. It points to a guild that the channel is part of.

MessageChannelLOG#

Description:

Since messages can send into multiple channels, each MessageLOG has multiple channels which cannot be stored inside the MessageLOG. This is why this table exists. It contains channels of each MessageLOG.

Attributes:

[Primary Key] [Foreign Key] log_id: Integer - Foreign key pointing to a row inside MessageLOG (to which log this channel log belongs to).
[Primary Key] [Foreign Key] channel_id: Integer - Foreign key pointing to a row inside the CHANNEL table.
reason: String - Reason why the send failed or NULL if send succeeded.

Custom Logger#

If you want to use a different logging scheme than the ones built in, you can do so by creating a custom logging manager that inherits the daf.logging.LoggerBASE.

The derived logger class can then implement the following methods:

__init__(self, param1, param2, …) [Required]:
The method used for passing parameters and for basic non-async initialization. This method must contain a fallback parameter and also needs to have an attribute of the same name.
Listing 10 Custom __init__ method#
class LoggerCUSTOM(daf.logging.LoggerBASE): def __init__(self, ..., logger): ... # Set attributes super().__init__(logger) ... # Other methods
async initialize(self) [Optional]:
The base’s initialize method calls initialize method of it’s fallback, if it fails then the fallback is set to None.

If you wish to do additional initialization that requires async/await operations, you can implement your own initialize method but make sure you call the base’s method in the end.
Listing 11 Custom initialize method#
class LoggerCUSTOM(daf.logging.LoggerBASE): ... # Other methods async def initialize(self): ... # Custom implementation code await super().initialize()
async _save_log(self, guild_context: dict, message_context: dict) [Required]:
Method that stores the message log. If there is any error in saving the log an exception should be raised, which will then make the logging module automatically use the fallback manager, do not call the fallback manager from this method!
Parameters:
guild_context (dict) - Contains keys:

“name”: The name of the guild/user (str)

“id”: Snowflake ID of the guild/user (int)

“type”: object type (GUILD/USER) that generated the log. (str)

message_context (dict) - Dictionary returned by:

daf.message.TextMESSAGE.generate_log_context()

daf.message.VoiceMESSAGE.generate_log_context()

daf.message.DirectMESSAGE.generate_log_context()
async update(self, **kwargs) [Optional]:
Custom implementation of the update method.

This method is used for updating the parameters that are available thru __init__ method and is not required if the attributes inside the object have the same name as the parameters inside the __init__ function and there are no pre-required steps that need to be taken before updating (see JSON Logging (file)’s code for example).

However if the name of attributes differ from parameter name or the attribute doesn’t exist at all or other steps are required than just re-initialization (see daf.logging.sql.LoggerSQL’s update method), then this method is required to be implemented. It should be implemented in a way that it calls the base update method. Example:
class LoggerCUSTOM(daf.logging.LoggerBASE): def __init__(self, name, fallback): self._name = name super().__init__(fallback) ... # Other methods async def update(self, **kwargs) # Only modify if the parameter is not passed to update method if "name" not kwargs: # The name parameter is stored under "_name" attribute instead of "name" kwargs["name"] = self._name ... # Other pre-required code (eg. remote SQL server needs to be disconnected) super().update(**kwargs) # Call base update method