Logging (core)#

Changed in version v2.7: Added invite link tracking

The logging module is responsible for 2 types of logging:

  1. Messages - Logs (attempts of) sent messages

  2. Invite links - Tracks new member joins with configured invite links.

Logging can be enabled for each GUILD / USER if the logging parameter is set to True.

Note

Invite links will be tracked regardless of the logging parameter. Invite link tracking is configured solely by the invite_track parameter inside GUILD.

Logging is handled thru so called logging managers and currently 3 exist:

Each logging manager can have a backup manager specified by it’s fallback parameter. If current manager fails, it’s fallback manager will be used temporarily to store the log. It will only use the fallback once and then, at the next logging attempt, the original manager will be used.

../../_images/logging_process.drawio.svg

Fig. 5 Logging process with fallback#

JSON Logging (file)#

The logs are written in the JSON format and saved into a JSON file, that has the name of the guild / user you were sending messages into. The JSON files are grouped by day and stored into folder Year/Month/Day, this means that each day a new JSON file will be generated for that specific day for easier managing, for example, if today is 13.07.2022, the log will be saved into the file that is located in

History
└───2022
│   └───07
│       └───13
|           └─── #David's dungeon.json

JSON structure#

The log structure is the same for both USER and GUILD. All logs will contain keys:

  • “name”: The name of the guild/user

  • “id”: Snowflake ID of the guild/user

  • “type”: object type (GUILD/USER) that generated the log.

  • “invite_tracking”: Dictionary that holds invite link tracking information.

    It’s keys are invite link ID’s (final part of invite link URL) and the value is a list of invite link logs, where a new log is created on each member join.

    Each invite log is a dictionary and contains the following keys:

    • “id”: Member’s snowflake (Discord) ID,

    • “name”: Member’s username,

    • “index”: serial number of the log,

    • “timestamp”: Date-Time when the log was created.

  • “message_tracking”: Dictionary that holds information about sent messages.

    Note

    Only messages sent from DAF are tracked. Other messages are not tracked.

    The keys are snowflake IDs of each each account who has sent the message from DAF.

    The value under each key is a dictionary containing:

JSON code example#

Listing 10 Code to produce JSON logs#
# Import the necessary items
from daf.logging import LoggerJSON
from daf.guild import GUILD
from datetime import timedelta
from daf.client import ACCOUNT
from daf.message.base import AutoCHANNEL
from daf.message.text_based import TextMESSAGE
from daf.logging.tracing import TraceLEVELS
import daf


rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]


@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item


# Define the logger
logger = LoggerJSON(
    path="History",
)

# Defined accounts
accounts = [
    ACCOUNT(
        token="OTA5MzgyNDE3MDg3ODgxMjc2.YZDeXw.34V-TbQSsxJxx8Fu399Mafu8jDI",
        is_user=False,
        intents=None,
        proxy=None,
        servers=[
            GUILD(
                snowflake=863071397207212052,
                messages=[
                    TextMESSAGE(
                        start_period=None,
                        end_period=timedelta(
                            seconds=5.0,
                        ),
                        data=get(rolls),
                        channels=AutoCHANNEL(
                            include_pattern="test",
                        ),
                    ),
                ],
                logging=True,
                remove_after=None,
                invite_track=[
                    "5fYEEpak",
                    "WxWdjKMp",
                    "qDvbRF7C",
                ],
            ),
        ],
    ),
]

# Run the framework (blocking)
daf.run(
    accounts=accounts,
    logger=logger,
    debug=TraceLEVELS.NORMAL
)

CSV Logging (file)#

The logs are written in the CSV format and saved into a CSV file, that has the name of the guild or an user you were sending messages into. The CSV files are fragmented by day and stored into folder Year/Month/Day, this means that each day a new CSV file will be generated for that specific day for easier managing, for example, if today is 13.07.2023, the log will be saved into the file that is located in

History
└───2023
│   └───07
│       └───13
|           └─── #David's dungeon.csv

CSV structure#

Warning

Invite link tracking is not supported with CSV logging.

The structure contains the following attributes:

  • Timestamp (string)

  • Guild Type (string),

  • Guild Name (string),

  • Guild Snowflake (integer),

  • Author name (string),

  • Author Snowflake (integer),

  • Message Type (string),

  • Sent Data (json),

  • Message Mode (non-empty for TextMESSAGE and DirectMESSAGE) (string),

  • Message Channels (non-empty for TextMESSAGE and VoiceMESSAGE) (json),

  • Success Info (non-empty for DirectMESSAGE) (json),

Note

Attributes marked with (json) are the same as in JSON Logging (file)

CSV code example#

Listing 11 Code to produce JSON logs#
# Import the necessary items
from daf.logging import LoggerCSV
from daf.guild import GUILD
from datetime import timedelta
from daf.client import ACCOUNT
from daf.message.base import AutoCHANNEL
from daf.message.text_based import TextMESSAGE
from daf.logging.tracing import TraceLEVELS
import daf


rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]


@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item


# Define the logger
logger = LoggerCSV(
    path="History",
    delimiter=';'
)

# Defined accounts
accounts = [
    ACCOUNT(
        token="OTA5MzgyNDE3MDg3ODgxMjc2.YZDeXw.34V-TbQSsxJxx8Fu399Mafu8jDI",
        is_user=False,
        intents=None,
        proxy=None,
        servers=[
            GUILD(
                snowflake=863071397207212052,
                messages=[
                    TextMESSAGE(
                        start_period=None,
                        end_period=timedelta(
                            seconds=5.0,
                        ),
                        data=get(rolls),
                        channels=AutoCHANNEL(
                            include_pattern="test",
                        ),
                    ),
                ],
                logging=True,
                remove_after=None,
                invite_track=[
                    "5fYEEpak",
                    "WxWdjKMp",
                    "qDvbRF7C",
                ],
            ),
        ],
    ),
]

# Run the framework (blocking)
daf.run(
    accounts=accounts,
    logger=logger,
    debug=TraceLEVELS.NORMAL
)

Relational Database Log (SQL)#

This type of logging enables saving logs to a remote server inside the database. In addition to being smaller in size, database logging takes up less space and it allows easier data analysis.

Dialects#

The dialect is selected via the dialect parameter in LoggerSQL. The following dialects are supported:

  • Microsoft SQL Server

  • PostgreSQL

  • SQLite,

  • MySQL

Usage#

For daf to use SQL logging, you need to pass the run() function with the logger parameter and pass it the LoggerSQL object.

# Import the necessary items
from daf.logging import LoggerSQL, LoggerJSON
from daf.guild import GUILD
from datetime import timedelta
from daf.client import ACCOUNT
from daf.message.base import AutoCHANNEL
from daf.message.text_based import TextMESSAGE
from daf.logging.tracing import TraceLEVELS
import daf


rolls = [
    "https://i.pinimg.com/originals/b7/fb/80/b7fb80122cf46d0e584f3a0768aef282.gif",
    "https://bit.ly/3sHrjQZ",
    "https://static.wikia.nocookie.net/a1dea591-8a10-4c02-a573-5321c601c129",
    "https://www.gifcen.com/wp-content/uploads/2022/03/rickroll-gif-4.gif",
    "https://bit.ly/3u5D8Dt",
    "http://static1.squarespace.com/static/60503ac20951e15087fbe7b8/60504609ee9c445722c9dd4e/60e3f9b541eb1b01e8e46854/1627103366283/RalphRoll.gif?format=1500w",
    "https://i.imgflip.com/56bhvt.gif",
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
]


@daf.data_function
def get(st):
    item = st.pop(0)
    st.append(item)
    return item


# Define the logger
logger = LoggerSQL(
    dialect="mysql",
    database="mydatabase",
    fallback=LoggerJSON(path="History")
)

# Defined accounts
accounts = [
    ACCOUNT(
        token="OTA5MzgyNDE3MDg3ODgxMjc2.YZDeXw.34V-TbQSsxJxx8Fu399Mafu8jDI",
        is_user=False,
        intents=None,
        proxy=None,
        servers=[
            GUILD(
                snowflake=863071397207212052,
                messages=[
                    TextMESSAGE(
                        start_period=None,
                        end_period=timedelta(
                            seconds=5.0,
                        ),
                        data=get(rolls),
                        channels=AutoCHANNEL(
                            include_pattern="test",
                        ),
                    ),
                ],
                logging=True,
                remove_after=None,
                invite_track=[
                    "5fYEEpak",
                    "WxWdjKMp",
                    "qDvbRF7C",
                ],
            ),
        ],
    ),
]

# Run the framework (blocking)
daf.run(
    accounts=accounts,
    logger=logger,
    debug=TraceLEVELS.NORMAL
)

Features#

  • Multiple dialects (sqlite, mssql, postgresql, mysql)

  • Automatic creation of the schema

  • Caching for faster logging

  • Low redundancy for reduced file size

  • Automatic error recovery

Warning

The database must already exist (unless using SQLite). However it can be completely empty, no need to manually create the schema.

ER diagram#

../../_images/sql_er.drawio.svg

Analysis#

The LoggerSQL provides some methods for data analysis:

SQL Tables#

MessageLOG#

Description:

This table contains the actual logs of sent messages, if the message type is DirectMESSAGE, then all the information is stored in this table. If the types are Voice/Text MESSAGE, then channel part of the log is saved in the MessageChannelLOG table.

Attributes:
  • [Primary Key] id: Integer - This is an internal ID of the log inside the database.

  • sent_data: Integer - Foreign key pointing to a row inside the DataHISTORY table.

  • message_type: SmallInteger - Foreign key ID pointing to a entry inside the MessageTYPE table.

  • guild_id: Integer - Foreign key pointing to GuildUSER table, represents guild id of guild the message was sent into.

  • author_id: Integer - Foreign key pointing to GuildUSER table, represents the author account of the message.

  • message_mode: SmallInteger - Foreign key pointing to MessageMODE table. This is non-null only for DirectMESSAGE.

  • dm_reason: String - If MessageTYPE is not DirectMESSAGE or the send attempt was successful, this is NULL, otherwise it contains the string representation of the error that caused the message send attempt to be unsuccessful.

  • timestamp: DateTime - The timestamp of the message send attempt.

DataHISTORY#

Description:

This table contains all the different data that was ever advertised. Every element is unique and is not replicated. This table exist to reduce redundancy and file size of the logs whenever same data is advertised multiple times. When a log is created, it is first checked if the data sent was already sent before, if it was the id to the existing DataHISTORY row is used, else a new row is created.

Attributes:
  • [Primary Key] id: Integer - Internal ID of data inside the database.

  • content: JSON - Actual data that was sent.

MessageTYPE#

Description:

This is a lookup table containing the the different message types that exist within the framework (Messages).

Attributes:
  • [Primary Key] id: SmallInteger - Internal ID of the message type inside the database.

  • name: String - The name of the actual message type.

GuildUSER#

Description:

The table contains all the guilds/users the framework ever generated a log for and all the authors.

Attributes:
  • [Primary Key] id: Integer - Internal ID of the Guild/User inside the database.

  • snowflake_id: BigInteger - The discord (snowflake) ID of the User/Guild

  • name: String - Name of the Guild/User

  • guild_type: SmallInteger - Foreign key pointing to GuildTYPE table.

MessageMODE#

Description:

This is a lookup table containing the the different message modes available by TextMESSAGE / DirectMESSAGE, it is set to null for VoiceMESSAGE.

Attributes:
  • [Primary Key] id: SmallInteger - Internal identifier of the message mode inside the database.

  • name: String - The name of the actual message mode.

GuildTYPE#

Description:

This is a lookup table containing types of the guilds inside the framework (Guilds).

Attributes:
  • [Primary Key] id: SmallInteger - Internal identifier of the guild type inside the database.

  • name: String - The name of the guild type.

CHANNEL#

Description:

The table contains all the channels that the framework ever advertised into.

Attributes:
  • [Primary Key] id: Integer - Internal identifier of the channel inside the database

  • snowflake_id: BigInteger - The discord (snowflake) identifier representing specific channel

  • name: String - The name of the channel

  • guild_id: Integer - Foreign key pointing to a row inside the GuildUSER table. It points to a guild that the channel is part of.

MessageChannelLOG#

Description:

Since messages can send into multiple channels, each MessageLOG has multiple channels which cannot be stored inside the MessageLOG. This is why this table exists. It contains channels of each MessageLOG.

Attributes:
  • [Primary Key] [Foreign Key] log_id: Integer - Foreign key pointing to a row inside MessageLOG (to which log this channel log belongs to).

  • [Primary Key] [Foreign Key] channel_id: Integer - Foreign key pointing to a row inside the CHANNEL table.

  • reason: String - Reason why the send failed or NULL if send succeeded.

Invite#

Description:

Table that represents tracked invite links.

Attributes:
  • [Primary Key] id: Integer - Internal ID of the invite inside the database.

  • [Foreign Key] guild_id: Integer - Foreign key pointing to a row inside the GuildUSER table (The guild that owns the invite).

  • discord_id: String - Discord’s invite ID (final part of the invite URL).

InviteLOG#

Description:

Table which’s entries are logs of member joins into a guild using a specific invite link.

Attributes:
  • [Primary Key] id: Integer - Internal ID of the log inside the database.

  • [Foreign Key] invite_id: Integer - Foreign key pointing to a row inside the Invite table. Describes the link member used to join a guild.

  • [Foreign Key] member_id: Integer - Foreign key pointing to a row inside the GuildUSER table. Describes the member who joined.

  • timestamp: DateTime - The date and time a member joined into a guild.