Studio (#4237)
* Rebuild Studio branch on top of main * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix security and code quality issues for Studio PR #4237 - Validate models_dir query param against allowed directory roots to prevent path traversal in /api/models/local endpoint - Replace string startswith() with Path.is_relative_to() for frontend path traversal check in serve_frontend - Sanitize SSE error messages to not leak exception details to clients (4 locations in inference.py) - Bind port-discovery socket to 127.0.0.1 instead of all interfaces in llama_cpp backend - Import datasets_root and resolve_output_dir in embedding training function to fix NameError and use managed output directory - Remove stale .gitignore entries for package-lock.json and test directories so tests can be tracked in version control - Add venv-reexecution logic to ui CLI command matching the studio command behavior * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Move models_dir path validation before try/except block The HTTPException(403) was inside the try/except Exception handler, so it would be caught and re-raised as a 500. Moving the validation before the try block ensures the 403 is returned directly and also makes the control flow clearer for static analysis (path is validated before any filesystem operations). * Use os.path.realpath + startswith for models_dir validation CodeQL py/path-injection does not recognize Path.is_relative_to() as a sanitizer. Switched to os.path.realpath + str.startswith which is a recognized sanitizer pattern in CodeQL's taint analysis. The startswith check uses root_str + os.sep to prevent prefix collisions (e.g. /app/models_evil matching /app/models). * Never pass user input to Path constructor in models_dir validation CodeQL traces taint through Path(resolved) even after a startswith barrier guard. Fix: the user-supplied models_dir is only used as a string for comparison against allowed roots. The Path object passed to _scan_models_dir comes from the trusted allowed_roots list, not from user input. This fully breaks the taint chain. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han
GitHub
parent
bced78373f
commit
f08aef1804
664 changed files with 103567 additions and 4 deletions
661
studio/LICENSE.AGPL-3.0
Normal file
661
studio/LICENSE.AGPL-3.0
Normal file
|
|
@ -0,0 +1,661 @@
|
|||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
111
studio/Unsloth_Studio_Colab.ipynb
Normal file
111
studio/Unsloth_Studio_Colab.ipynb
Normal file
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f2b0c6a1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**License Notice**\n",
|
||||
"\n",
|
||||
"SPDX-License-Identifier: AGPL-3.0-only\n",
|
||||
"\n",
|
||||
"Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "447c1156",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ===========================================\n",
|
||||
"# ⚠️ GPU Check - Run This First!\n",
|
||||
"# ===========================================\n",
|
||||
"import torch\n",
|
||||
"\n",
|
||||
"print(\"🔍 Checking for GPU...\")\n",
|
||||
"if not torch.cuda.is_available():\n",
|
||||
" print(\"❌ ERROR: No GPU detected!\")\n",
|
||||
" print(\"\\n📋 To enable GPU:\")\n",
|
||||
" print(\" 1. Go to: Runtime → Change runtime type\")\n",
|
||||
" print(\" 2. Select: Hardware accelerator → GPU (T4 is free)\")\n",
|
||||
" print(\" 3. Click: Save\")\n",
|
||||
" print(\" 4. Restart and re-run all cells\")\n",
|
||||
" raise RuntimeError(\"⛔ GPU required for Unsloth Studio\")\n",
|
||||
"else:\n",
|
||||
" gpu_name = torch.cuda.get_device_name(0)\n",
|
||||
" print(f\"✅ GPU detected: {gpu_name}\")\n",
|
||||
" print(\" Ready to proceed!\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "f04a9b46",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ===========================================\n",
|
||||
"# GitHub Authentication (Private Repo)\n",
|
||||
"# ===========================================\n",
|
||||
"from getpass import getpass\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"print(\"🔐 GitHub Token Required\")\n",
|
||||
"print(\"Get token: https://github.com/settings/tokens\")\n",
|
||||
"print(\"Scope needed: 'repo'\")\n",
|
||||
"print(\"-\" * 50)\n",
|
||||
"\n",
|
||||
"github_token = getpass(\"Enter GitHub Token: \")\n",
|
||||
"os.environ['GITHUB_TOKEN'] = github_token\n",
|
||||
"print(\"✅ Token stored\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "27e68f91",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ===========================================\n",
|
||||
"# Setup: Clone repo and run setup\n",
|
||||
"# ===========================================\n",
|
||||
"\n",
|
||||
"import os\n",
|
||||
"github_token = os.environ['GITHUB_TOKEN']\n",
|
||||
"!git clone https://{github_token}@github.com/unslothai/new-ui-prototype.git\n",
|
||||
"%cd /content/new-ui-prototype\n",
|
||||
"\n",
|
||||
"# Run setup script\n",
|
||||
"!chmod +x setup.sh\n",
|
||||
"!./setup.sh"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "277e431e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# ===========================================\n",
|
||||
"# Start Unsloth Studio\n",
|
||||
"# ===========================================\n",
|
||||
"import sys\n",
|
||||
"sys.path.insert(0, '/content/new-ui-prototype/studio/backend')\n",
|
||||
"\n",
|
||||
"from colab import start\n",
|
||||
"start()"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
2
studio/__init__.py
Normal file
2
studio/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
2
studio/backend/__init__.py
Normal file
2
studio/backend/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
2
studio/backend/assets/__init__.py
Normal file
2
studio/backend/assets/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
2
studio/backend/assets/configs/__init__.py
Normal file
2
studio/backend/assets/configs/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
42
studio/backend/assets/configs/full_finetune.yaml
Normal file
42
studio/backend/assets/configs/full_finetune.yaml
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
model: unsloth/Qwen2.5-0.5B
|
||||
|
||||
data:
|
||||
dataset: tatsu-lab/alpaca
|
||||
format_type: auto
|
||||
|
||||
training:
|
||||
training_type: full
|
||||
max_seq_length: 2048
|
||||
load_in_4bit: false
|
||||
output_dir: outputs
|
||||
num_epochs: 1
|
||||
learning_rate: 0.0002
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 0
|
||||
save_steps: 0
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: "unsloth"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules: ""
|
||||
vision_all_linear: false
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: unsloth-training
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: runs
|
||||
42
studio/backend/assets/configs/lora_text.yaml
Normal file
42
studio/backend/assets/configs/lora_text.yaml
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
model: unsloth/Qwen2.5-0.5B
|
||||
|
||||
data:
|
||||
dataset: tatsu-lab/alpaca
|
||||
format_type: auto
|
||||
|
||||
training:
|
||||
training_type: lora
|
||||
max_seq_length: 2048
|
||||
load_in_4bit: true
|
||||
output_dir: outputs
|
||||
num_epochs: 1
|
||||
learning_rate: 0.0002
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 0
|
||||
save_steps: 0
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: "unsloth"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules: "q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj"
|
||||
vision_all_linear: false
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: unsloth-training
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: runs
|
||||
56
studio/backend/assets/configs/model_defaults/default.yaml
Normal file
56
studio/backend/assets/configs/model_defaults/default.yaml
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
# Default model training parameters
|
||||
# Used for models without specific configurations
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 5e-5
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_ratio: 0.1
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.7
|
||||
top_p: 0.95
|
||||
top_k: -1
|
||||
min_p: 0.01
|
||||
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
# Model defaults for unsloth/Qwen3-Embedding-0.6B
|
||||
# Based on Qwen3_Embedding_(0_6B).py embedding notebook
|
||||
# Also applies to: unsloth/Qwen3-Embedding-4B
|
||||
|
||||
training:
|
||||
max_seq_length: 512
|
||||
# num_epochs: 2
|
||||
num_epochs: 0
|
||||
learning_rate: 3e-5
|
||||
batch_size: 256
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: false
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "constant_with_warmup"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "embedding-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 50
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# Model defaults for unsloth/all-MiniLM-L6-v2
|
||||
# Based on All_MiniLM_L6_v2.py embedding notebook
|
||||
|
||||
training:
|
||||
max_seq_length: 512
|
||||
# num_epochs: 2
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 256
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: false
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "value"
|
||||
- "key"
|
||||
- "dense"
|
||||
- "query"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "embedding-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 50
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# Model defaults for unsloth/bge-m3
|
||||
# Based on BGE_M3.py embedding notebook
|
||||
|
||||
training:
|
||||
max_seq_length: 512
|
||||
# num_epochs: 2
|
||||
num_epochs: 0
|
||||
learning_rate: 3e-5
|
||||
batch_size: 256
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: false
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "constant_with_warmup"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "key"
|
||||
- "query"
|
||||
- "dense"
|
||||
- "value"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "embedding-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 50
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
# Model defaults for unsloth/embeddinggemma-300m
|
||||
# Based on EmbeddingGemma_(300M).py embedding notebook
|
||||
|
||||
training:
|
||||
max_seq_length: 1024
|
||||
# num_epochs: 1
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-5
|
||||
batch_size: 64
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "embedding-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 5
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
# Model defaults for unsloth/gte-modernbert-base
|
||||
# Based on ModernBert.py embedding notebook
|
||||
|
||||
training:
|
||||
max_seq_length: 512
|
||||
# num_epochs: 2
|
||||
num_epochs: 0
|
||||
learning_rate: 3e-5
|
||||
batch_size: 256
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "constant_with_warmup"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "Wi"
|
||||
- "Wo"
|
||||
- "Wqkv"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "embedding-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 50
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/ERNIE-4.5-21B-A3B-PT
|
||||
# Based on ERNIE_4_5_21B_A3B_PT-Conversational.ipynb
|
||||
# Also applies to: unsloth/ERNIE-4.5-21B-A3B-PT
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# Model defaults for unsloth/ERNIE-4.5-VL-28B-A3B-PT
|
||||
# Based on ERNIE_4_5_VL_28B_A3B_PT_Vision.ipynb
|
||||
# Also applies to: unsloth/ERNIE-4.5-VL-28B-A3B-PT
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: true
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: true
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for tiiuae/Falcon-H1-0.5B-Instruct
|
||||
# Based on Falcon_H1_(0.5B)-Alpaca.ipynb
|
||||
# Also applies to: tiiuae/Falcon-H1-0.5B-Instruct, unsloth/Falcon-H1-0.5B-Instruct
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 8
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: false
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.1
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# Model defaults for unsloth/codegemma-7b-bnb-4bit
|
||||
# Based on CodeGemma_(7B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/codegemma-7b, google/codegemma-7b
|
||||
# added inference parameters from Ollama
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 4096
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0
|
||||
top_p: 0.9
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/functiongemma-270m-it
|
||||
# Based on FunctionGemma_(270M).ipynb
|
||||
# Also applies to: unsloth/functiongemma-270m-it-unsloth-bnb-4bit, google/functiongemma-270m-it, unsloth/functiongemma-270m-it-unsloth-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 4096
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 10
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 128
|
||||
lora_alpha: 256
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# Model defaults for unsloth/gemma-2-27b-bnb-4bit
|
||||
# Based on Gemma2_(9B)-Alpaca.ipynb (same defaults for larger models)
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/gemma-2-2b
|
||||
# Based on Gemma2_(2B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/gemma-2-2b-bnb-4bit, google/gemma-2-2b
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/gemma-3-270m-it
|
||||
# Based on Gemma3_(270M).ipynb
|
||||
# Also applies to: unsloth/gemma-3-270m-it-unsloth-bnb-4bit, google/gemma-3-270m-it, unsloth/gemma-3-270m-it-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 5e-5
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 128
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/gemma-3-27b-it
|
||||
# Based on Gemma3_(27B)_A100-Conversational.ipynb
|
||||
# Also applies to: unsloth/gemma-3-27b-it-unsloth-bnb-4bit, google/gemma-3-27b-it, unsloth/gemma-3-27b-it-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 8
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/gemma-3-4b-it
|
||||
# Based on Gemma3_(4B).ipynb
|
||||
# Also applies to: unsloth/gemma-3-4b-it-unsloth-bnb-4bit, google/gemma-3-4b-it, unsloth/gemma-3-4b-it-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 8
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/gemma-3-4b-pt
|
||||
# Based on Gemma3_(4B)-Vision.ipynb
|
||||
# Also applies to: unsloth/gemma-3-4b-pt-unsloth-bnb-4bit, google/gemma-3-4b-pt, unsloth/gemma-3-4b-pt-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 2
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: true
|
||||
optim: "adamw_torch_fused"
|
||||
lr_scheduler_type: "cosine"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/gemma-3n-E4B-it
|
||||
# Based on Gemma3N_(4B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit, google/gemma-3n-E4B-it, unsloth/gemma-3n-E4B-it-unsloth-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 1024
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 8
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
audio_input: true
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/gemma-3n-E4B
|
||||
# Based on Gemma3N_(4B)-Vision.ipynb
|
||||
# Also applies to: unsloth/gemma-3n-E4B-unsloth-bnb-4bit, google/gemma-3n-E4B
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 2
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_ratio: 0.03
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: true
|
||||
optim: "adamw_torch_fused"
|
||||
lr_scheduler_type: "cosine"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
audio_input: true
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_k: 64
|
||||
top_p: 0.95
|
||||
min_p: 0.0
|
||||
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Model defaults for unsloth/gpt-oss-120b
|
||||
# Based on gpt-oss-(120B)_A100-Fine-tuning.ipynb
|
||||
# Also applies to: openai/gpt-oss-120b, unsloth/gpt-oss-120b-unsloth-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 4096
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_p: 1.0
|
||||
top_k: 0
|
||||
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Model defaults for unsloth/gpt-oss-20b
|
||||
# Based on gpt-oss-(20B)-Fine-tuning.ipynb
|
||||
# Also applies to: openai/gpt-oss-20b, unsloth/gpt-oss-20b-unsloth-bnb-4bit, unsloth/gpt-oss-20b-BF16
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 1024
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.0
|
||||
top_p: 1.0
|
||||
top_k: 0
|
||||
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# Model defaults for unsloth/granite-4.0-350m
|
||||
# Based on Granite4.0_350M.ipynb
|
||||
# Also applies to: ibm-granite/granite-4.0-350m, unsloth/granite-4.0-350m-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
- "shared_mlp.input_linear"
|
||||
- "shared_mlp.output_linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 0
|
||||
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# Model defaults for unsloth/granite-4.0-h-micro
|
||||
# Based on Granite4.0.ipynb
|
||||
# Also applies to: ibm-granite/granite-4.0-h-micro, unsloth/granite-4.0-h-micro-bnb-4bit, unsloth/granite-4.0-h-micro-unsloth-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
- "shared_mlp.input_linear"
|
||||
- "shared_mlp.output_linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.0
|
||||
top_p: 1.0
|
||||
top_k: 0
|
||||
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Model defaults for unsloth/Llama-3.2-11B-Vision-Instruct
|
||||
# Based on Llama3.2_(11B)-Vision.ipynb
|
||||
# Also applies to: unsloth/Llama-3.2-11B-Vision-Instruct-unsloth-bnb-4bit, meta-llama/Llama-3.2-11B-Vision-Instruct, unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Llama-3.2-1B-Instruct
|
||||
# Based on Llama3.2_(1B)-RAFT.ipynb
|
||||
# Also applies to: unsloth/Llama-3.2-1B-Instruct-unsloth-bnb-4bit, meta-llama/Llama-3.2-1B-Instruct, unsloth/Llama-3.2-1B-Instruct-bnb-4bit, RedHatAI/Llama-3.2-1B-Instruct-FP8, unsloth/Llama-3.2-1B-Instruct-FP8-Block, unsloth/Llama-3.2-1B-Instruct-FP8-Dynamic
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 5
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-5
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 8
|
||||
warmup_steps: 0
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: true
|
||||
optim: "adamw_torch"
|
||||
lr_scheduler_type: "cosine"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/Llama-3.2-3B-Instruct
|
||||
# Based on Llama3.2_(1B_and_3B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/Llama-3.2-3B-Instruct-unsloth-bnb-4bit, meta-llama/Llama-3.2-3B-Instruct, unsloth/Llama-3.2-3B-Instruct-bnb-4bit, RedHatAI/Llama-3.2-3B-Instruct-FP8, unsloth/Llama-3.2-3B-Instruct-FP8-Block, unsloth/Llama-3.2-3B-Instruct-FP8-Dynamic
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/Llama-3.3-70B-Instruct
|
||||
# Based on Llama3.3_(70B)_A100-Conversational.ipynb
|
||||
# Also applies to: unsloth/Llama-3.3-70B-Instruct-unsloth-bnb-4bit, meta-llama/Llama-3.3-70B-Instruct, unsloth/Llama-3.3-70B-Instruct-bnb-4bit, RedHatAI/Llama-3.3-70B-Instruct-FP8, unsloth/Llama-3.3-70B-Instruct-FP8-Block, unsloth/Llama-3.3-70B-Instruct-FP8-Dynamic
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Meta-Llama-3.1-70B-bnb-4bit
|
||||
# Based on Llama3.1_(8B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/Meta-Llama-3.1-8B-bnb-4bit, unsloth/Meta-Llama-3.1-8B-unsloth-bnb-4bit, meta-llama/Meta-Llama-3.1-8B, unsloth/Meta-Llama-3.1-8B, unsloth/Meta-Llama-3.1-70B, meta-llama/Meta-Llama-3.1-70B, unsloth/Meta-Llama-3.1-405B-bnb-4bit, meta-llama/Meta-Llama-3.1-405B
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
|
||||
# Based on Llama3.1_(8B)-Inference.ipynb
|
||||
# Also applies to: "unsloth/Meta-Llama-3.1-8B-Instruct-unsloth-bnb-4bit", "meta-llama/Meta-Llama-3.1-8B-Instruct", "unsloth/Meta-Llama-3.1-8B-Instruct","RedHatAI/Llama-3.1-8B-Instruct-FP8","unsloth/Llama-3.1-8B-Instruct-FP8-Block","unsloth/Llama-3.1-8B-Instruct-FP8-Dynamic"
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 8192
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/llama-3-8b-Instruct-bnb-4bit
|
||||
# Based on Llama3_(8B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/llama-3-8b-Instruct, meta-llama/Meta-Llama-3-8B-Instruct
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/llama-3-8b-bnb-4bit
|
||||
# Based on Llama3_(8B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/llama-3-8b, meta-llama/Meta-Llama-3-8B
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# Model defaults for unsloth/Llasa-3B
|
||||
# Based on Llasa_TTS_(3B).ipynb and Llasa_TTS_(1B).ipynb
|
||||
# Also applies to: HKUSTAudio/Llasa-1B
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 5e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 128
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "v_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.2
|
||||
top_p: 1.2
|
||||
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
# Model defaults for unsloth/Magistral-Small-2509
|
||||
# Based on Magistral_(24B)-Reasoning-Conversational.ipynb
|
||||
# Also applies to: mistralai/Magistral-Small-2509, unsloth/Magistral-Small-2509-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.7
|
||||
min_p: 0.01
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# Model defaults for unsloth/Ministral-3-3B-Instruct-2512
|
||||
# Based on Ministral_3_VL_(3B)_Vision.ipynb
|
||||
# Also applies to: unsloth/Ministral-3-3B-Instruct-2512
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.15
|
||||
top_p: default
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Mistral-Nemo-Base-2407-bnb-4bit
|
||||
# Based on Mistral_Nemo_(12B)-Alpaca.ipynb
|
||||
# Also applies to: "unsloth/Mistral-Nemo-Base-2407", "mistralai/Mistral-Nemo-Base-2407", "unsloth/Mistral-Nemo-Instruct-2407-bnb-4bit", "unsloth/Mistral-Nemo-Instruct-2407", "mistralai/Mistral-Nemo-Instruct-2407",
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Mistral-Small-Instruct-2409
|
||||
# Based on Mistral_Small_(22B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/Mistral-Small-Instruct-2409-bnb-4bit, mistralai/Mistral-Small-Instruct-2409
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Model defaults for unsloth/Pixtral-12B-2409
|
||||
# Based on Pixtral_(12B)-Vision.ipynb
|
||||
# Also applies to: unsloth/Pixtral-12B-2409-unsloth-bnb-4bit, mistralai/Pixtral-12B-2409, unsloth/Pixtral-12B-2409-bnb-4bit
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "paged_adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 8
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: false
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/mistral-7b-instruct-v0.3-bnb-4bit
|
||||
# Based on Mistral_v0.3_(7B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/mistral-7b-instruct-v0.3, mistralai/Mistral-7B-Instruct-v0.3
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# Model defaults for unsloth/mistral-7b-v0.3-bnb-4bit
|
||||
# Based on Mistral_v0.3_(7B)-Alpaca.ipynb
|
||||
# Also applies to: "unsloth/mistral-7b-v0.3", "mistralai/Mistral-7B-v0.3",
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for OuteAI/Llama-OuteTTS-1.0-1B
|
||||
# Based on Oute_TTS_(1B).ipynb
|
||||
# Also applies to: OuteAI/Llama-OuteTTS-1.0-1B
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
audio_type: dac
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
eval_steps: 0
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 128
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "v_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.4
|
||||
top_k: 40
|
||||
top_p: 0.9
|
||||
min_p: 0.05
|
||||
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# Model defaults for Spark-TTS-0.5B/LLM
|
||||
# Based on Spark_TTS_(0_5B).ipynb
|
||||
# Also applies to: Spark-TTS-0.5B/LLM
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
audio_type: bicodec
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
eval_steps: 0
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 128
|
||||
lora_alpha: 128
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.8
|
||||
top_k: 50
|
||||
top_p: 1.0
|
||||
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# Model defaults for sesame/csm-1b
|
||||
# Based on Sesame_CSM_(1B)-TTS.ipynb
|
||||
# Also applies to: sesame/csm-1b
|
||||
|
||||
audio_type: csm
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
eval_steps: 0
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Model defaults for unsloth/GLM-4.7-Flash
|
||||
# Based on GLM_Flash_A100(80GB).py
|
||||
# Also applies to: unsloth/GLM-4.7-Flash-unsloth-bnb-4bit, unsloth/GLM-4.7-Flash-bnb-4bit, THUDM/GLM-4.7-Flash
|
||||
|
||||
training:
|
||||
trust_remote_code: true
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 60
|
||||
save_steps: 60
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
- "out_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: true
|
||||
temperature: 0.7
|
||||
top_p: 0.8
|
||||
top_k: 20
|
||||
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
# Model defaults for unsloth/LFM2-1.2B
|
||||
# Based on Liquid_LFM2_(1.2B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/LFM2-1.2B
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.3
|
||||
min_p: 0.15
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/Nemotron-3-Nano-30B-A3B
|
||||
# Based on Nemotron-3-Nano-30B-A3B_A100.ipynb
|
||||
# Also applies to: unsloth/Nemotron-3-Nano-30B-A3B
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: true
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 8
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
- "in_proj"
|
||||
- "out_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: true
|
||||
temperature: 1.0
|
||||
top_p: 1.0
|
||||
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# Model defaults for unsloth/PaddleOCR-VL
|
||||
# Based on Paddle_OCR_(1B)_Vision.ipynb
|
||||
# Also applies to: unsloth/PaddleOCR-VL
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: true
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 5e-5
|
||||
batch_size: 4
|
||||
gradient_accumulation_steps: 2
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: true
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# Model defaults for answerdotai/ModernBERT-large
|
||||
# Based on bert_classification.ipynb
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 1
|
||||
num_epochs: 0
|
||||
learning_rate: 5e-5
|
||||
batch_size: 32
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# Model defaults for unsloth/orpheus-3b-0.1-ft
|
||||
# Based on Orpheus_(3B)-TTS.ipynb
|
||||
# Also applies to: unsloth/orpheus-3b-0.1-ft-unsloth-bnb-4bit, canopylabs/orpheus-3b-0.1-ft, unsloth/orpheus-3b-0.1-ft-bnb-4bit
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
audio_type: snac
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
eval_steps: 0
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/tinyllama
|
||||
# Based on TinyLlama_(1.1B)-Alpaca.ipynb
|
||||
# Also applies to: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 4096
|
||||
# num_epochs: 1
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-5
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_ratio: 0.1
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.1
|
||||
random_seed: 3407
|
||||
packing: true
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# Model defaults for unsloth/whisper-large-v3
|
||||
# Based on Whisper.ipynb
|
||||
# Also applies to: unsloth/whisper-large-v3, openai/whisper-large-v3
|
||||
|
||||
audio_type: whisper
|
||||
audio_input: true
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
eval_steps: 5
|
||||
max_seq_length: 448
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 1e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "v_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Phi-3-medium-4k-instruct
|
||||
# Based on Phi_3_Medium-Conversational.ipynb
|
||||
# Also applies to: "unsloth/Phi-3-medium-4k-instruct-bnb-4bit", "microsoft/Phi-3-medium-4k-instruct",
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Phi-3.5-mini-instruct
|
||||
# Based on Phi_3.5_Mini-Conversational.ipynb
|
||||
# Also applies to: "unsloth/Phi-3.5-mini-instruct-bnb-4bit", "microsoft/Phi-3.5-mini-instruct"
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/Phi-4
|
||||
# Based on Phi_4-Conversational.ipynb
|
||||
# Also applies to: unsloth/phi-4-unsloth-bnb-4bit, microsoft/phi-4, unsloth/phi-4-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.8
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for imdatta0/tiny_qwen3_moe_2.8B_0.7B
|
||||
# Based on TinyQwen3_MoE.py
|
||||
# Dummy model of qwen3moe architecture created to fit in T4
|
||||
# MoE model - includes gate_up_proj for MoE layers
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_steps: 5
|
||||
max_steps: 50
|
||||
save_steps: 50
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
- "gate_up_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Qwen2-7B
|
||||
# Based on Qwen2_(7B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/Qwen2-7B-bnb-4bit, Qwen/Qwen2-7B
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Model defaults for unsloth/Qwen2-VL-7B-Instruct
|
||||
# Based on Qwen2_VL_(7B)-Vision.ipynb
|
||||
# Also applies to: unsloth/Qwen2-VL-7B-Instruct-unsloth-bnb-4bit, Qwen/Qwen2-VL-7B-Instruct, unsloth/Qwen2-VL-7B-Instruct-bnb-4bit
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Qwen2.5-1.5B-Instruct
|
||||
# Based on nemo_gym_sudoku.ipynb
|
||||
# Also applies to: unsloth/Qwen2.5-1.5B-Instruct-unsloth-bnb-4bit, Qwen/Qwen2.5-1.5B-Instruct, unsloth/Qwen2.5-1.5B-Instruct-bnb-4bit
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 4096
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 1e-5
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 64
|
||||
warmup_ratio: 0.1
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 42
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 4
|
||||
lora_alpha: 8
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Qwen2.5-7B
|
||||
# Based on Qwen2.5_(7B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/Qwen2.5-7B-unsloth-bnb-4bit, Qwen/Qwen2.5-7B, unsloth/Qwen2.5-7B-bnb-4bit
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Qwen2.5-Coder-1.5B-Instruct
|
||||
# Based on Qwen2.5_Coder_(1.5B)-Tool_Calling.ipynb
|
||||
# Also applies to: unsloth/Qwen2.5-Coder-1.5B-Instruct-bnb-4bit, Qwen/Qwen2.5-Coder-1.5B-Instruct
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/Qwen2.5-Coder-14B-Instruct
|
||||
# Based on Qwen2.5_Coder_(14B)-Conversational.ipynb
|
||||
# Also applies to: unsloth/Qwen2.5-Coder-14B-Instruct-bnb-4bit, Qwen/Qwen2.5-Coder-14B-Instruct
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "paged_adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Model defaults for unsloth/Qwen2.5-Coder-7B-Instruct-bnb-4bit
|
||||
# Based on CodeForces-cot-Finetune_for_Reasoning_on_CodeForces.ipynb
|
||||
# Also applies to: unsloth/Qwen2.5-Coder-7B-Instruct, Qwen/Qwen2.5-Coder-7B-Instruct
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 32768
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Model defaults for unsloth/Qwen2.5-VL-7B-Instruct-bnb-4bit
|
||||
# Based on Qwen2.5_VL_(7B)-Vision.ipynb
|
||||
# Also applies to: unsloth/Qwen2.5-VL-7B-Instruct, Qwen/Qwen2.5-VL-7B-Instruct, unsloth/Qwen2.5-VL-7B-Instruct-unsloth-bnb-4bit
|
||||
# added inference parameters from unsloth notebook
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 1.5
|
||||
min_p: 0.1
|
||||
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Model defaults for unsloth/Qwen3-0.6B
|
||||
# Based on Qwen3_(0_6B)-Phone_Deployment.ipynb
|
||||
# Also applies to: unsloth/Qwen3-0.6B-unsloth-bnb-4bit, Qwen/Qwen3-0.6B, unsloth/Qwen3-0.6B-bnb-4bit, Qwen/Qwen3-0.6B-FP8, unsloth/Qwen3-0.6B-FP8
|
||||
# added inference parameters from Ollama
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 1024
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 5e-5
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
# Model defaults for unsloth/Qwen3-14B-Base
|
||||
# Based on Qwen3_(14B)-Alpaca.ipynb
|
||||
# Also applies to: unsloth/Qwen3-14B-Base, Qwen/Qwen3-14B-Base, unsloth/Qwen3-14B-Base-bnb-4bit
|
||||
# added inference parameters from Ollama
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Model defaults for unsloth/Qwen3-14B
|
||||
# Based on Qwen3_(14B).ipynb
|
||||
# Also applies to: unsloth/Qwen3-14B-unsloth-bnb-4bit, Qwen/Qwen3-14B, unsloth/Qwen3-14B-bnb-4bit, Qwen/Qwen3-14B-FP8, unsloth/Qwen3-14B-FP8
|
||||
# added inference parameters from Ollama
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/Qwen3-30B-A3B-Instruct-2507
|
||||
# Based on Qwen3_MoE.py
|
||||
# Also applies to: Qwen/Qwen3-30B-A3B-Instruct-2507, unsloth/Qwen3-30B-A3B-Instruct-2507-bnb-4bit
|
||||
# MoE model - includes gate_up_proj for MoE layers
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 1
|
||||
warmup_steps: 5
|
||||
max_steps: 50
|
||||
save_steps: 50
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 64
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
- "gate_up_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Model defaults for unsloth/Qwen3-32B
|
||||
# Based on Qwen3_(32B)_A100-Reasoning-Conversational.ipynb
|
||||
# Also applies to: unsloth/Qwen3-32B-unsloth-bnb-4bit, Qwen/Qwen3-32B, unsloth/Qwen3-32B-bnb-4bit, Qwen/Qwen3-32B-FP8, unsloth/Qwen3-32B-FP8
|
||||
# added inference parameters from Ollama
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_k: 20
|
||||
top_p: 0.95
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/Qwen3-4B-Instruct-2507
|
||||
# Based on Qwen3_(4B)-Instruct.ipynb
|
||||
# Also applies to: unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit, Qwen/Qwen3-4B-Instruct-2507, unsloth/Qwen3-4B-Instruct-2507-bnb-4bit, Qwen/Qwen3-4B-Instruct-2507-FP8, unsloth/Qwen3-4B-Instruct-2507-FP8
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.7
|
||||
top_p: 0.80
|
||||
top_k: 20
|
||||
min_p: 0.00
|
||||
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Model defaults for unsloth/Qwen3-4B-Thinking-2507
|
||||
# Based on Qwen3_(4B)-Thinking.ipynb
|
||||
# Also applies to: unsloth/Qwen3-4B-Thinking-2507-unsloth-bnb-4bit, Qwen/Qwen3-4B-Thinking-2507, unsloth/Qwen3-4B-Thinking-2507-bnb-4bit, Qwen/Qwen3-4B-Thinking-2507-FP8, unsloth/Qwen3-4B-Thinking-2507-FP8
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 32
|
||||
lora_alpha: 32
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "q_proj"
|
||||
- "k_proj"
|
||||
- "v_proj"
|
||||
- "o_proj"
|
||||
- "gate_proj"
|
||||
- "up_proj"
|
||||
- "down_proj"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.6
|
||||
top_p: 0.95
|
||||
top_k: 20
|
||||
min_p: 0.00
|
||||
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# Model defaults for unsloth/Qwen3-VL-8B-Instruct
|
||||
# Based on Qwen3_VL_(8B)-Vision.ipynb
|
||||
# Also applies to: Qwen/Qwen3-VL-8B-Instruct-FP8, unsloth/Qwen3-VL-8B-Instruct-FP8, unsloth/Qwen3-VL-8B-Instruct, Qwen/Qwen3-VL-8B-Instruct, unsloth/Qwen3-VL-8B-Instruct-bnb-4bit
|
||||
# added inference parameters from unsloth guides
|
||||
|
||||
training:
|
||||
trust_remote_code: false
|
||||
max_seq_length: 2048
|
||||
# num_epochs: 4
|
||||
num_epochs: 0
|
||||
learning_rate: 2e-4
|
||||
batch_size: 2
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 30
|
||||
save_steps: 30
|
||||
weight_decay: 0.001
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: true
|
||||
gradient_checkpointing: "unsloth"
|
||||
optim: "adamw_8bit"
|
||||
lr_scheduler_type: "linear"
|
||||
|
||||
lora:
|
||||
lora_r: 16
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules:
|
||||
- "all-linear"
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: "llm-finetuning"
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: "runs"
|
||||
log_frequency: 10
|
||||
|
||||
inference:
|
||||
trust_remote_code: false
|
||||
temperature: 0.7
|
||||
top_p: 0.8
|
||||
top_k: 20
|
||||
|
||||
42
studio/backend/assets/configs/vision_lora.yaml
Normal file
42
studio/backend/assets/configs/vision_lora.yaml
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
model: unsloth/Qwen2-VL-2B-Instruct-bnb-4bit
|
||||
|
||||
data:
|
||||
dataset: philschmid/amazon-product-descriptions-vlm
|
||||
format_type: auto
|
||||
|
||||
training:
|
||||
training_type: lora
|
||||
max_seq_length: 2048
|
||||
load_in_4bit: true
|
||||
output_dir: outputs
|
||||
num_epochs: 1
|
||||
learning_rate: 0.0002
|
||||
batch_size: 1
|
||||
gradient_accumulation_steps: 4
|
||||
warmup_steps: 5
|
||||
max_steps: 0
|
||||
save_steps: 0
|
||||
weight_decay: 0.01
|
||||
random_seed: 3407
|
||||
packing: false
|
||||
train_on_completions: false
|
||||
gradient_checkpointing: "unsloth"
|
||||
|
||||
lora:
|
||||
lora_r: 64
|
||||
lora_alpha: 16
|
||||
lora_dropout: 0.0
|
||||
target_modules: "" # vision uses vision_all_linear by default
|
||||
vision_all_linear: true
|
||||
use_rslora: false
|
||||
use_loftq: false
|
||||
finetune_vision_layers: true
|
||||
finetune_language_layers: true
|
||||
finetune_attention_modules: true
|
||||
finetune_mlp_modules: true
|
||||
|
||||
logging:
|
||||
enable_wandb: false
|
||||
wandb_project: unsloth-training
|
||||
enable_tensorboard: false
|
||||
tensorboard_dir: runs
|
||||
1288
studio/backend/assets/datasets/alpaca_unsloth.json
Normal file
1288
studio/backend/assets/datasets/alpaca_unsloth.json
Normal file
File diff suppressed because it is too large
Load diff
0
studio/backend/auth/.gitkeep
Normal file
0
studio/backend/auth/.gitkeep
Normal file
47
studio/backend/auth/__init__.py
Normal file
47
studio/backend/auth/__init__.py
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""
|
||||
Authentication module for JWT-based auth with SQLite storage.
|
||||
"""
|
||||
|
||||
from .authentication import (
|
||||
create_access_token,
|
||||
create_refresh_token,
|
||||
refresh_access_token,
|
||||
get_current_subject,
|
||||
reload_secret,
|
||||
)
|
||||
from .storage import (
|
||||
is_initialized,
|
||||
create_initial_user,
|
||||
get_user_and_secret,
|
||||
load_jwt_secret,
|
||||
save_setup_token,
|
||||
consume_setup_token,
|
||||
has_pending_setup_token,
|
||||
save_refresh_token,
|
||||
verify_refresh_token,
|
||||
revoke_user_refresh_tokens,
|
||||
)
|
||||
from .hashing import hash_password, verify_password
|
||||
|
||||
__all__ = [
|
||||
"create_access_token",
|
||||
"create_refresh_token",
|
||||
"refresh_access_token",
|
||||
"get_current_subject",
|
||||
"reload_secret",
|
||||
"is_initialized",
|
||||
"create_initial_user",
|
||||
"get_user_and_secret",
|
||||
"load_jwt_secret",
|
||||
"save_setup_token",
|
||||
"consume_setup_token",
|
||||
"has_pending_setup_token",
|
||||
"save_refresh_token",
|
||||
"verify_refresh_token",
|
||||
"revoke_user_refresh_tokens",
|
||||
"hash_password",
|
||||
"verify_password",
|
||||
]
|
||||
108
studio/backend/auth/authentication.py
Normal file
108
studio/backend/auth/authentication.py
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||
import jwt
|
||||
|
||||
from .storage import load_jwt_secret, save_refresh_token, verify_refresh_token
|
||||
|
||||
ALGORITHM = "HS256"
|
||||
ACCESS_TOKEN_EXPIRE_MINUTES = 60
|
||||
REFRESH_TOKEN_EXPIRE_DAYS = 7
|
||||
|
||||
# Load stable secret from SQLite (set during first-time setup)
|
||||
# This will raise RuntimeError if auth hasn't been initialized yet
|
||||
try:
|
||||
SECRET_KEY = load_jwt_secret()
|
||||
except RuntimeError:
|
||||
# Fallback: use a temporary secret until setup is complete
|
||||
# This allows the app to start, but protected routes will fail until setup
|
||||
SECRET_KEY = secrets.token_urlsafe(64)
|
||||
|
||||
security = HTTPBearer() # Reads Authorization: Bearer <token>
|
||||
|
||||
|
||||
def create_access_token(
|
||||
subject: str,
|
||||
expires_delta: Optional[timedelta] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Create a signed JWT for the given subject (e.g. username).
|
||||
|
||||
Tokens are valid across restarts because SECRET_KEY is stored in SQLite.
|
||||
"""
|
||||
to_encode = {"sub": subject}
|
||||
expire = datetime.now(timezone.utc) + (
|
||||
expires_delta or timedelta(minutes = ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||
)
|
||||
to_encode.update({"exp": expire})
|
||||
return jwt.encode(to_encode, SECRET_KEY, algorithm = ALGORITHM)
|
||||
|
||||
|
||||
def create_refresh_token(subject: str) -> str:
|
||||
"""
|
||||
Create a random refresh token, store its hash in SQLite, and return it.
|
||||
|
||||
Refresh tokens are opaque (not JWTs) and expire after REFRESH_TOKEN_EXPIRE_DAYS.
|
||||
"""
|
||||
token = secrets.token_urlsafe(48)
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(days = REFRESH_TOKEN_EXPIRE_DAYS)
|
||||
save_refresh_token(token, subject, expires_at.isoformat())
|
||||
return token
|
||||
|
||||
|
||||
def refresh_access_token(refresh_token: str) -> Optional[str]:
|
||||
"""
|
||||
Validate a refresh token and issue a new access token.
|
||||
|
||||
The refresh token itself is NOT consumed — it stays valid until expiry.
|
||||
Returns a new access_token or None if the refresh token is invalid/expired.
|
||||
"""
|
||||
username = verify_refresh_token(refresh_token)
|
||||
if username is None:
|
||||
return None
|
||||
return create_access_token(subject = username)
|
||||
|
||||
|
||||
def reload_secret() -> None:
|
||||
"""
|
||||
Reload the JWT secret from SQLite.
|
||||
|
||||
Call this after setup to ensure new tokens use the persistent secret.
|
||||
"""
|
||||
global SECRET_KEY
|
||||
SECRET_KEY = load_jwt_secret()
|
||||
|
||||
|
||||
async def get_current_subject(
|
||||
credentials: HTTPAuthorizationCredentials = Depends(security),
|
||||
) -> str:
|
||||
"""
|
||||
FastAPI dependency to validate the JWT and return the subject.
|
||||
|
||||
Use this as a dependency on routes that should be protected, e.g.:
|
||||
|
||||
@router.get("/secure")
|
||||
async def secure_endpoint(current_subject: str = Depends(get_current_subject)):
|
||||
...
|
||||
"""
|
||||
token = credentials.credentials
|
||||
try:
|
||||
payload = jwt.decode(token, SECRET_KEY, algorithms = [ALGORITHM])
|
||||
subject: Optional[str] = payload.get("sub")
|
||||
if subject is None:
|
||||
raise HTTPException(
|
||||
status_code = status.HTTP_401_UNAUTHORIZED,
|
||||
detail = "Invalid token payload",
|
||||
)
|
||||
return subject
|
||||
except jwt.InvalidTokenError:
|
||||
raise HTTPException(
|
||||
status_code = status.HTTP_401_UNAUTHORIZED,
|
||||
detail = "Invalid or expired token",
|
||||
)
|
||||
43
studio/backend/auth/hashing.py
Normal file
43
studio/backend/auth/hashing.py
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""
|
||||
Password hashing utilities using PBKDF2.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import secrets
|
||||
from typing import Tuple
|
||||
|
||||
|
||||
def hash_password(password: str, salt: str | None = None) -> Tuple[str, str]:
|
||||
"""
|
||||
Hash a password using PBKDF2-HMAC-SHA256.
|
||||
|
||||
Returns (salt, hex_hash) tuple.
|
||||
"""
|
||||
if salt is None:
|
||||
salt = secrets.token_hex(16)
|
||||
dk = hashlib.pbkdf2_hmac(
|
||||
"sha256",
|
||||
password.encode("utf-8"),
|
||||
salt.encode("utf-8"),
|
||||
100_000, # 100k iterations
|
||||
)
|
||||
return salt, dk.hex()
|
||||
|
||||
|
||||
def verify_password(password: str, salt: str, hashed: str) -> bool:
|
||||
"""
|
||||
Verify a password against a stored salt and hash.
|
||||
|
||||
Uses constant-time comparison to prevent timing attacks.
|
||||
"""
|
||||
dk = hashlib.pbkdf2_hmac(
|
||||
"sha256",
|
||||
password.encode("utf-8"),
|
||||
salt.encode("utf-8"),
|
||||
100_000,
|
||||
)
|
||||
return hmac.compare_digest(dk.hex(), hashed)
|
||||
263
studio/backend/auth/storage.py
Normal file
263
studio/backend/auth/storage.py
Normal file
|
|
@ -0,0 +1,263 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""
|
||||
SQLite storage for authentication data (user credentials + JWT secret).
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from utils.paths import auth_db_path, ensure_dir
|
||||
|
||||
DB_PATH = auth_db_path()
|
||||
|
||||
|
||||
def _hash_token(token: str) -> str:
|
||||
"""SHA-256 hash a setup token for safe storage."""
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def get_connection() -> sqlite3.Connection:
|
||||
"""Get a connection to the auth database, creating tables if needed."""
|
||||
ensure_dir(DB_PATH.parent)
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS auth_user (
|
||||
id INTEGER PRIMARY KEY,
|
||||
username TEXT UNIQUE NOT NULL,
|
||||
password_salt TEXT NOT NULL,
|
||||
password_hash TEXT NOT NULL,
|
||||
jwt_secret TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS setup_tokens (
|
||||
id INTEGER PRIMARY KEY,
|
||||
token_hash TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS refresh_tokens (
|
||||
id INTEGER PRIMARY KEY,
|
||||
token_hash TEXT NOT NULL,
|
||||
username TEXT NOT NULL,
|
||||
expires_at TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
)
|
||||
conn.commit()
|
||||
return conn
|
||||
|
||||
|
||||
def is_initialized() -> bool:
|
||||
"""Check if auth has been set up (user exists in DB)."""
|
||||
conn = get_connection()
|
||||
cur = conn.execute("SELECT COUNT(*) AS c FROM auth_user")
|
||||
row = cur.fetchone()
|
||||
conn.close()
|
||||
return bool(row["c"])
|
||||
|
||||
|
||||
def create_initial_user(username: str, password: str, jwt_secret: str) -> None:
|
||||
"""
|
||||
Create the initial admin user in the database.
|
||||
|
||||
Raises sqlite3.IntegrityError if username already exists.
|
||||
"""
|
||||
from .hashing import hash_password
|
||||
|
||||
salt, pwd_hash = hash_password(password)
|
||||
conn = get_connection()
|
||||
try:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO auth_user (username, password_salt, password_hash, jwt_secret)
|
||||
VALUES (?, ?, ?, ?)
|
||||
""",
|
||||
(username, salt, pwd_hash, jwt_secret),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def delete_user(username: str) -> None:
|
||||
"""
|
||||
Delete a user from the database.
|
||||
|
||||
Used for rollback when setup fails after user creation.
|
||||
"""
|
||||
conn = get_connection()
|
||||
try:
|
||||
conn.execute("DELETE FROM auth_user WHERE username = ?", (username,))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_user_and_secret(username: str) -> Optional[Tuple[str, str, str]]:
|
||||
"""
|
||||
Get user's password salt, hash, and JWT secret.
|
||||
|
||||
Returns (password_salt, password_hash, jwt_secret) or None if user not found.
|
||||
"""
|
||||
conn = get_connection()
|
||||
try:
|
||||
cur = conn.execute(
|
||||
"""
|
||||
SELECT password_salt, password_hash, jwt_secret
|
||||
FROM auth_user
|
||||
WHERE username = ?
|
||||
""",
|
||||
(username,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
return None
|
||||
return row["password_salt"], row["password_hash"], row["jwt_secret"]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_jwt_secret() -> str:
|
||||
"""
|
||||
Load the JWT secret from the database.
|
||||
|
||||
Raises RuntimeError if auth is not initialized.
|
||||
"""
|
||||
conn = get_connection()
|
||||
try:
|
||||
cur = conn.execute("SELECT jwt_secret FROM auth_user LIMIT 1")
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
raise RuntimeError(
|
||||
"Auth is not initialized. Please set up a password first."
|
||||
)
|
||||
return row["jwt_secret"]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def save_setup_token(token: str) -> None:
|
||||
"""
|
||||
Store a hashed setup token, replacing any existing one.
|
||||
"""
|
||||
token_hash = _hash_token(token)
|
||||
conn = get_connection()
|
||||
try:
|
||||
conn.execute("DELETE FROM setup_tokens")
|
||||
conn.execute("INSERT INTO setup_tokens (token_hash) VALUES (?)", (token_hash,))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def consume_setup_token(token: str) -> bool:
|
||||
"""
|
||||
Verify a setup token and delete it if valid.
|
||||
|
||||
Returns True if the token was valid (and is now consumed), False otherwise.
|
||||
"""
|
||||
token_hash = _hash_token(token)
|
||||
conn = get_connection()
|
||||
try:
|
||||
cur = conn.execute(
|
||||
"SELECT id FROM setup_tokens WHERE token_hash = ?", (token_hash,)
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
conn.execute("DELETE FROM setup_tokens WHERE id = ?", (row["id"],))
|
||||
conn.commit()
|
||||
return True
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def has_pending_setup_token() -> bool:
|
||||
"""Check if a setup token is waiting to be consumed."""
|
||||
conn = get_connection()
|
||||
try:
|
||||
cur = conn.execute("SELECT COUNT(*) AS c FROM setup_tokens")
|
||||
row = cur.fetchone()
|
||||
return bool(row["c"])
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def save_refresh_token(token: str, username: str, expires_at: str) -> None:
|
||||
"""
|
||||
Store a hashed refresh token with its associated username and expiry.
|
||||
"""
|
||||
token_hash = _hash_token(token)
|
||||
conn = get_connection()
|
||||
try:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO refresh_tokens (token_hash, username, expires_at)
|
||||
VALUES (?, ?, ?)
|
||||
""",
|
||||
(token_hash, username, expires_at),
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def verify_refresh_token(token: str) -> Optional[str]:
|
||||
"""
|
||||
Verify a refresh token and return the username.
|
||||
|
||||
Returns the username if valid and not expired, None otherwise.
|
||||
The token is NOT consumed — it stays valid until it expires.
|
||||
"""
|
||||
token_hash = _hash_token(token)
|
||||
conn = get_connection()
|
||||
try:
|
||||
# Clean up any expired tokens while we're here
|
||||
conn.execute(
|
||||
"DELETE FROM refresh_tokens WHERE expires_at < ?",
|
||||
(datetime.now(timezone.utc).isoformat(),),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
cur = conn.execute(
|
||||
"""
|
||||
SELECT id, username, expires_at FROM refresh_tokens
|
||||
WHERE token_hash = ?
|
||||
""",
|
||||
(token_hash,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
# Check expiry
|
||||
expires_at = datetime.fromisoformat(row["expires_at"])
|
||||
if datetime.now(timezone.utc) > expires_at:
|
||||
conn.execute("DELETE FROM refresh_tokens WHERE id = ?", (row["id"],))
|
||||
conn.commit()
|
||||
return None
|
||||
|
||||
return row["username"]
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def revoke_user_refresh_tokens(username: str) -> None:
|
||||
"""Revoke all refresh tokens for a user (e.g. on logout)."""
|
||||
conn = get_connection()
|
||||
try:
|
||||
conn.execute("DELETE FROM refresh_tokens WHERE username = ?", (username,))
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
99
studio/backend/colab.py
Normal file
99
studio/backend/colab.py
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""
|
||||
Colab-specific helpers for running Unsloth Studio.
|
||||
Uses Colab's built-in proxy - no external tunneling needed!
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
# Add backend to path early so local modules like loggers can be imported
|
||||
backend_path = str(Path(__file__).parent)
|
||||
if backend_path not in sys.path:
|
||||
sys.path.insert(0, backend_path)
|
||||
|
||||
from loggers import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
def get_colab_url(port: int = 8000) -> str:
|
||||
"""
|
||||
Get the actual Colab proxy URL for a port.
|
||||
"""
|
||||
try:
|
||||
from google.colab.output import eval_js
|
||||
|
||||
# Use Colab's proxy mechanism
|
||||
url = eval_js(f"google.colab.kernel.proxyPort({port})", timeout_sec = 5)
|
||||
return url if url else f"http://localhost:{port}"
|
||||
except Exception as e:
|
||||
logger.info(f"Note: Could not get Colab URL ({e})")
|
||||
return f"http://localhost:{port}"
|
||||
|
||||
|
||||
def show_link(port: int = 8000):
|
||||
"""Display a styled clickable link to the UI."""
|
||||
from IPython.display import display, HTML
|
||||
|
||||
# Get real Colab proxy URL
|
||||
url = get_colab_url(port)
|
||||
|
||||
html = f"""
|
||||
<div style="padding: 20px; background: linear-gradient(135deg, #22c55e 0%, #16a34a 100%);
|
||||
border-radius: 12px; margin: 10px 0; font-family: system-ui, -apple-system, sans-serif;">
|
||||
<h2 style="color: white; margin: 0 0 12px 0; font-size: 24px;">
|
||||
🦥 Unsloth Studio is Ready!
|
||||
</h2>
|
||||
<a href="{url}" target="_blank"
|
||||
style="display: inline-block; padding: 14px 28px; background: white; color: #16a34a;
|
||||
text-decoration: none; border-radius: 8px; font-weight: 600; font-size: 16px;
|
||||
box-shadow: 0 4px 6px rgba(0,0,0,0.1);">
|
||||
🚀 Open Unsloth Studio
|
||||
</a>
|
||||
<p style="color: rgba(255,255,255,0.9); margin: 16px 0 0 0; font-size: 13px;
|
||||
word-break: break-all; font-family: monospace;">
|
||||
{url}
|
||||
</p>
|
||||
</div>
|
||||
"""
|
||||
display(HTML(html))
|
||||
|
||||
|
||||
def start(port: int = 8000):
|
||||
"""
|
||||
Start Unsloth Studio server in Colab and display the URL.
|
||||
|
||||
Usage:
|
||||
from colab import start
|
||||
start()
|
||||
"""
|
||||
import sys
|
||||
|
||||
logger.info("🦥 Starting Unsloth Studio...")
|
||||
|
||||
logger.info(" Loading backend...")
|
||||
from run import run_server
|
||||
|
||||
# Auto-detect frontend path
|
||||
repo_root = Path(__file__).parent.parent
|
||||
frontend_path = repo_root / "frontend" / "dist"
|
||||
|
||||
if not frontend_path.exists():
|
||||
logger.info("❌ Frontend not built! Please run the setup cell first.")
|
||||
return
|
||||
|
||||
logger.info(" Starting server...")
|
||||
# Start server silently
|
||||
run_server(host = "0.0.0.0", port = port, frontend_path = frontend_path, silent = True)
|
||||
|
||||
logger.info(" Server started!")
|
||||
|
||||
# Show the clickable link with real URL
|
||||
show_link(port)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
start()
|
||||
134
studio/backend/core/__init__.py
Normal file
134
studio/backend/core/__init__.py
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""
|
||||
Unified core module for Unsloth backend
|
||||
|
||||
Imports are LAZY (via __getattr__) so that training subprocesses can
|
||||
import core.training.worker without pulling in heavy ML dependencies
|
||||
like unsloth, transformers, or torch before the version activation
|
||||
code has a chance to run.
|
||||
"""
|
||||
|
||||
__all__ = [
|
||||
# Inference
|
||||
"InferenceBackend",
|
||||
"get_inference_backend",
|
||||
# Training
|
||||
"get_training_backend",
|
||||
"TrainingBackend",
|
||||
"TrainingProgress",
|
||||
# Config
|
||||
"ModelConfig",
|
||||
"is_vision_model",
|
||||
"scan_trained_loras",
|
||||
"load_model_defaults",
|
||||
"get_base_model_from_lora",
|
||||
# Utils
|
||||
"format_and_template_dataset",
|
||||
"normalize_path",
|
||||
"is_local_path",
|
||||
"is_model_cached",
|
||||
"without_hf_auth",
|
||||
"format_error_message",
|
||||
"get_gpu_memory_info",
|
||||
"log_gpu_memory",
|
||||
"get_device",
|
||||
"is_apple_silicon",
|
||||
"clear_gpu_cache",
|
||||
"DeviceType",
|
||||
]
|
||||
|
||||
|
||||
def __getattr__(name):
|
||||
# Inference
|
||||
if name in ("InferenceBackend", "get_inference_backend"):
|
||||
from .inference import InferenceBackend, get_inference_backend
|
||||
|
||||
globals()["InferenceBackend"] = InferenceBackend
|
||||
globals()["get_inference_backend"] = get_inference_backend
|
||||
return globals()[name]
|
||||
|
||||
# Training
|
||||
if name in ("TrainingBackend", "get_training_backend", "TrainingProgress"):
|
||||
from .training import TrainingBackend, get_training_backend, TrainingProgress
|
||||
|
||||
globals()["TrainingBackend"] = TrainingBackend
|
||||
globals()["get_training_backend"] = get_training_backend
|
||||
globals()["TrainingProgress"] = TrainingProgress
|
||||
return globals()[name]
|
||||
|
||||
# Config (from utils.models)
|
||||
if name in (
|
||||
"is_vision_model",
|
||||
"ModelConfig",
|
||||
"scan_trained_loras",
|
||||
"load_model_defaults",
|
||||
"get_base_model_from_lora",
|
||||
):
|
||||
from utils.models import (
|
||||
is_vision_model,
|
||||
ModelConfig,
|
||||
scan_trained_loras,
|
||||
load_model_defaults,
|
||||
get_base_model_from_lora,
|
||||
)
|
||||
|
||||
globals()["is_vision_model"] = is_vision_model
|
||||
globals()["ModelConfig"] = ModelConfig
|
||||
globals()["scan_trained_loras"] = scan_trained_loras
|
||||
globals()["load_model_defaults"] = load_model_defaults
|
||||
globals()["get_base_model_from_lora"] = get_base_model_from_lora
|
||||
return globals()[name]
|
||||
|
||||
# Paths
|
||||
if name in ("normalize_path", "is_local_path", "is_model_cached"):
|
||||
from utils.paths import normalize_path, is_local_path, is_model_cached
|
||||
|
||||
globals()["normalize_path"] = normalize_path
|
||||
globals()["is_local_path"] = is_local_path
|
||||
globals()["is_model_cached"] = is_model_cached
|
||||
return globals()[name]
|
||||
|
||||
# Utils
|
||||
if name in ("without_hf_auth", "format_error_message"):
|
||||
from utils.utils import without_hf_auth, format_error_message
|
||||
|
||||
globals()["without_hf_auth"] = without_hf_auth
|
||||
globals()["format_error_message"] = format_error_message
|
||||
return globals()[name]
|
||||
|
||||
# Hardware
|
||||
if name in (
|
||||
"get_device",
|
||||
"is_apple_silicon",
|
||||
"clear_gpu_cache",
|
||||
"get_gpu_memory_info",
|
||||
"log_gpu_memory",
|
||||
"DeviceType",
|
||||
):
|
||||
from utils.hardware import (
|
||||
get_device,
|
||||
is_apple_silicon,
|
||||
clear_gpu_cache,
|
||||
get_gpu_memory_info,
|
||||
log_gpu_memory,
|
||||
DeviceType,
|
||||
)
|
||||
|
||||
globals()["get_device"] = get_device
|
||||
globals()["is_apple_silicon"] = is_apple_silicon
|
||||
globals()["clear_gpu_cache"] = clear_gpu_cache
|
||||
globals()["get_gpu_memory_info"] = get_gpu_memory_info
|
||||
globals()["log_gpu_memory"] = log_gpu_memory
|
||||
globals()["DeviceType"] = DeviceType
|
||||
return globals()[name]
|
||||
|
||||
# Datasets
|
||||
if name == "format_and_template_dataset":
|
||||
from utils.datasets import format_and_template_dataset
|
||||
|
||||
globals()["format_and_template_dataset"] = format_and_template_dataset
|
||||
return format_and_template_dataset
|
||||
|
||||
raise AttributeError(f"module 'core' has no attribute {name!r}")
|
||||
10
studio/backend/core/data_recipe/__init__.py
Normal file
10
studio/backend/core/data_recipe/__init__.py
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
"""
|
||||
Data Recipe core (DataDesigner wrapper + job runner).
|
||||
"""
|
||||
|
||||
from .jobs import JobManager, get_job_manager
|
||||
|
||||
__all__ = ["JobManager", "get_job_manager"]
|
||||
6
studio/backend/core/data_recipe/jobs/__init__.py
Normal file
6
studio/backend/core/data_recipe/jobs/__init__.py
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from .manager import JobManager, get_job_manager
|
||||
|
||||
__all__ = ["JobManager", "get_job_manager"]
|
||||
33
studio/backend/core/data_recipe/jobs/constants.py
Normal file
33
studio/backend/core/data_recipe/jobs/constants.py
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# stages parsed from data-designer logs
|
||||
STAGE_CREATE = "create"
|
||||
STAGE_PREVIEW = "preview"
|
||||
STAGE_DAG = "dag"
|
||||
STAGE_HEALTHCHECK = "healthcheck"
|
||||
STAGE_SAMPLING = "sampling"
|
||||
STAGE_COLUMN_CONFIG = "column_config"
|
||||
STAGE_GENERATING = "generating"
|
||||
STAGE_BATCH = "batch"
|
||||
STAGE_PROFILING = "profiling"
|
||||
|
||||
USAGE_RESET_STAGES = {
|
||||
STAGE_CREATE,
|
||||
STAGE_PREVIEW,
|
||||
STAGE_DAG,
|
||||
STAGE_HEALTHCHECK,
|
||||
STAGE_SAMPLING,
|
||||
STAGE_GENERATING,
|
||||
STAGE_PROFILING,
|
||||
}
|
||||
|
||||
# job event types emitted by worker/manager
|
||||
EVENT_JOB_ENQUEUED = "job.enqueued"
|
||||
EVENT_JOB_STARTED = "job.started"
|
||||
EVENT_JOB_CANCELLING = "job.cancelling"
|
||||
EVENT_JOB_CANCELLED = "job.cancelled"
|
||||
EVENT_JOB_COMPLETED = "job.completed"
|
||||
EVENT_JOB_ERROR = "job.error"
|
||||
474
studio/backend/core/data_recipe/jobs/manager.py
Normal file
474
studio/backend/core/data_recipe/jobs/manager.py
Normal file
|
|
@ -0,0 +1,474 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import queue
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import multiprocessing as mp
|
||||
|
||||
from ..jsonable import to_preview_jsonable
|
||||
from .constants import (
|
||||
EVENT_JOB_CANCELLING,
|
||||
EVENT_JOB_CANCELLED,
|
||||
EVENT_JOB_COMPLETED,
|
||||
EVENT_JOB_ENQUEUED,
|
||||
EVENT_JOB_ERROR,
|
||||
EVENT_JOB_STARTED,
|
||||
)
|
||||
from .parse import apply_update, coerce_event, parse_log_message
|
||||
from .types import Job
|
||||
from .worker import run_job_process
|
||||
|
||||
|
||||
_CTX = mp.get_context("spawn")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Subscription:
|
||||
replay: list[dict]
|
||||
_q: queue.Queue
|
||||
_next_id: int = 0
|
||||
|
||||
async def next_event(self, *, timeout_sec: float) -> dict | None:
|
||||
"""Wait for next event (SSE), w/ timeout so we can check disconnects."""
|
||||
try:
|
||||
return await asyncio.to_thread(self._q.get, True, timeout_sec)
|
||||
except queue.Empty:
|
||||
return None
|
||||
|
||||
def format_sse(self, event: dict) -> bytes:
|
||||
"""Turn event dict into SSE bytes (id/event/data)."""
|
||||
event_id = event.get("seq")
|
||||
if event_id is None:
|
||||
self._next_id += 1
|
||||
event_id = self._next_id
|
||||
body = json.dumps(event, separators = (",", ":"), ensure_ascii = False)
|
||||
event_type = event.get("type") or "message"
|
||||
return (
|
||||
f"id: {event_id}\n" f"event: {event_type}\n" f"data: {body}\n\n"
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
class JobManager:
|
||||
def __init__(self) -> None:
|
||||
"""Single-job runner (in-mem). Simple on purpose, not a whole platform."""
|
||||
self._lock = threading.Lock()
|
||||
self._job: Job | None = None
|
||||
self._proc: mp.Process | None = None
|
||||
self._mp_q: Any | None = None
|
||||
self._events: deque[dict] = deque(maxlen = 5000)
|
||||
self._subs: list[queue.Queue] = []
|
||||
self._pump_thread: threading.Thread | None = None
|
||||
self._seq: int = 0
|
||||
|
||||
def start(self, *, recipe: dict, run: dict) -> str:
|
||||
"""Spawn the job subprocess (one at a time, no cap)."""
|
||||
llm_columns = recipe.get("columns") or []
|
||||
llm_column_count = 0
|
||||
if isinstance(llm_columns, list):
|
||||
for column in llm_columns:
|
||||
if not isinstance(column, dict):
|
||||
continue
|
||||
column_type = str(column.get("column_type") or "").strip().lower()
|
||||
if column_type.startswith("llm"):
|
||||
llm_column_count += 1
|
||||
if llm_column_count <= 0:
|
||||
llm_column_count = 1
|
||||
|
||||
with self._lock:
|
||||
if self._proc is not None and self._proc.is_alive():
|
||||
raise RuntimeError("job already running")
|
||||
|
||||
job_id = uuid.uuid4().hex
|
||||
self._job = Job(job_id = job_id, status = "pending", started_at = time.time())
|
||||
self._job.progress_columns_total = llm_column_count
|
||||
self._events.clear()
|
||||
self._seq = 0
|
||||
|
||||
run_payload = dict(run)
|
||||
run_payload["_job_id"] = job_id
|
||||
mp_q = _CTX.Queue()
|
||||
proc = _CTX.Process(
|
||||
target = run_job_process,
|
||||
kwargs = {"event_queue": mp_q, "recipe": recipe, "run": run_payload},
|
||||
daemon = True,
|
||||
)
|
||||
proc.start()
|
||||
|
||||
self._mp_q = mp_q
|
||||
self._proc = proc
|
||||
self._pump_thread = threading.Thread(target = self._pump_loop, daemon = True)
|
||||
self._pump_thread.start()
|
||||
|
||||
self._emit(
|
||||
{"type": EVENT_JOB_ENQUEUED, "ts": time.time(), "job_id": job_id}
|
||||
)
|
||||
return job_id
|
||||
|
||||
def cancel(self, job_id: str) -> bool:
|
||||
"""Hard stop. We terminate the subprocess. Quick + reliable."""
|
||||
with self._lock:
|
||||
if self._job is None or self._job.job_id != job_id:
|
||||
return False
|
||||
if self._proc is None or not self._proc.is_alive():
|
||||
return True
|
||||
self._job.status = "cancelling"
|
||||
self._emit(
|
||||
{"type": EVENT_JOB_CANCELLING, "ts": time.time(), "job_id": job_id}
|
||||
)
|
||||
try:
|
||||
self._proc.terminate()
|
||||
except (AttributeError, OSError):
|
||||
pass
|
||||
return True
|
||||
|
||||
def get_status(self, job_id: str) -> dict | None:
|
||||
"""UI friendly snapshot that we need. Alternative to sse kinda of and structured"""
|
||||
with self._lock:
|
||||
if self._job is None or self._job.job_id != job_id:
|
||||
return None
|
||||
job = self._job
|
||||
return {
|
||||
"job_id": job.job_id,
|
||||
"status": job.status,
|
||||
"stage": job.stage,
|
||||
"current_column": job.current_column,
|
||||
"completed_columns": list(job.completed_columns),
|
||||
"batch": {"idx": job.batch.idx, "total": job.batch.total},
|
||||
"progress": {
|
||||
"done": job.progress.done,
|
||||
"total": job.progress.total,
|
||||
"percent": job.progress.percent,
|
||||
"eta_sec": job.progress.eta_sec,
|
||||
"rate": job.progress.rate,
|
||||
"ok": job.progress.ok,
|
||||
"failed": job.progress.failed,
|
||||
},
|
||||
"column_progress": {
|
||||
"done": job.column_progress.done,
|
||||
"total": job.column_progress.total,
|
||||
"percent": job.column_progress.percent,
|
||||
"eta_sec": job.column_progress.eta_sec,
|
||||
"rate": job.column_progress.rate,
|
||||
"ok": job.column_progress.ok,
|
||||
"failed": job.column_progress.failed,
|
||||
},
|
||||
"model_usage": {
|
||||
name: {
|
||||
"model": usage.model,
|
||||
"tokens": {
|
||||
"input": usage.input_tokens,
|
||||
"output": usage.output_tokens,
|
||||
"total": usage.total_tokens,
|
||||
"tps": usage.tps,
|
||||
},
|
||||
"requests": {
|
||||
"success": usage.requests_success,
|
||||
"failed": usage.requests_failed,
|
||||
"total": usage.requests_total,
|
||||
"rpm": usage.rpm,
|
||||
},
|
||||
}
|
||||
for name, usage in job.model_usage.items()
|
||||
},
|
||||
"rows": job.rows,
|
||||
"cols": job.cols,
|
||||
"error": job.error,
|
||||
"has_analysis": job.analysis is not None,
|
||||
"dataset_rows": None if job.dataset is None else len(job.dataset),
|
||||
"artifact_path": job.artifact_path,
|
||||
"started_at": job.started_at,
|
||||
"finished_at": job.finished_at,
|
||||
}
|
||||
|
||||
def get_current_status(self) -> dict | None:
|
||||
"""Single-job convenience (last/current)."""
|
||||
job_id = self.get_current_job_id()
|
||||
if job_id is None:
|
||||
return None
|
||||
return self.get_status(job_id)
|
||||
|
||||
def get_current_job_id(self) -> str | None:
|
||||
"""Return current job_id (or None)."""
|
||||
with self._lock:
|
||||
return None if self._job is None else self._job.job_id
|
||||
|
||||
def get_analysis(self, job_id: str) -> dict | None:
|
||||
"""Final profiling output (only after job completes)."""
|
||||
with self._lock:
|
||||
if self._job is None or self._job.job_id != job_id:
|
||||
return None
|
||||
return self._job.analysis
|
||||
|
||||
def get_dataset(
|
||||
self,
|
||||
job_id: str,
|
||||
*,
|
||||
limit: int,
|
||||
offset: int = 0,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Load dataset page (offset + limit) and include total rows."""
|
||||
with self._lock:
|
||||
if self._job is None or self._job.job_id != job_id:
|
||||
return None
|
||||
in_memory_dataset = self._job.dataset
|
||||
artifact_path = self._job.artifact_path
|
||||
job_status = self._job.status
|
||||
|
||||
if in_memory_dataset is not None:
|
||||
total = len(in_memory_dataset)
|
||||
rows = in_memory_dataset[offset : offset + limit]
|
||||
return {"dataset": rows, "total": total}
|
||||
if not artifact_path:
|
||||
if job_status in {"completed", "error", "cancelled"}:
|
||||
return {"error": "artifact path missing"}
|
||||
return None
|
||||
|
||||
try:
|
||||
base_dataset_path = Path(artifact_path)
|
||||
parquet_dir = base_dataset_path / "parquet-files"
|
||||
if not parquet_dir.exists():
|
||||
return {"error": f"dataset path missing: {parquet_dir}"}
|
||||
|
||||
return self._load_dataset_page(
|
||||
parquet_dir = parquet_dir, limit = limit, offset = offset
|
||||
)
|
||||
except Exception as exc:
|
||||
return {"error": f"dataset load failed: {exc}"}
|
||||
|
||||
@staticmethod
|
||||
def _load_dataset_page(
|
||||
*,
|
||||
parquet_dir: Path,
|
||||
limit: int,
|
||||
offset: int,
|
||||
) -> dict[str, Any]:
|
||||
dataset_page = JobManager._load_dataset_page_with_duckdb(
|
||||
parquet_dir = parquet_dir,
|
||||
limit = limit,
|
||||
offset = offset,
|
||||
)
|
||||
if dataset_page is not None:
|
||||
return dataset_page
|
||||
return JobManager._load_dataset_page_with_data_designer(
|
||||
parquet_dir = parquet_dir,
|
||||
limit = limit,
|
||||
offset = offset,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _load_dataset_page_with_duckdb(
|
||||
*,
|
||||
parquet_dir: Path,
|
||||
limit: int,
|
||||
offset: int,
|
||||
) -> dict[str, Any] | None:
|
||||
parquet_glob = str((parquet_dir / "*.parquet").resolve())
|
||||
try:
|
||||
import duckdb # type: ignore
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
try:
|
||||
conn = duckdb.connect(":memory:")
|
||||
try:
|
||||
total_row = conn.execute(
|
||||
"SELECT COUNT(*) FROM read_parquet(?)",
|
||||
[parquet_glob],
|
||||
).fetchone()
|
||||
total = int(total_row[0] if total_row else 0)
|
||||
dataframe = conn.execute(
|
||||
(
|
||||
"SELECT *, row_number() OVER (PARTITION BY filename) AS __row_num__ "
|
||||
"FROM read_parquet(?, filename=true) "
|
||||
"ORDER BY filename, __row_num__ "
|
||||
"LIMIT ? OFFSET ?"
|
||||
),
|
||||
[parquet_glob, int(limit), int(offset)],
|
||||
).fetchdf()
|
||||
finally:
|
||||
conn.close()
|
||||
except (RuntimeError, ValueError, duckdb.Error):
|
||||
return None
|
||||
|
||||
for helper_col in ("filename", "__row_num__"):
|
||||
if helper_col in dataframe.columns:
|
||||
dataframe = dataframe.drop(columns = [helper_col])
|
||||
|
||||
rows = dataframe.to_dict(orient = "records")
|
||||
return {"dataset": to_preview_jsonable(rows), "total": total}
|
||||
|
||||
@staticmethod
|
||||
def _load_dataset_page_with_data_designer(
|
||||
*,
|
||||
parquet_dir: Path,
|
||||
limit: int,
|
||||
offset: int,
|
||||
) -> dict[str, Any]:
|
||||
from data_designer.config.utils.io_helpers import read_parquet_dataset
|
||||
|
||||
dataframe = read_parquet_dataset(parquet_dir)
|
||||
total = int(len(dataframe.index))
|
||||
rows = dataframe.iloc[offset : offset + limit].to_dict(orient = "records")
|
||||
return {"dataset": to_preview_jsonable(rows), "total": total}
|
||||
|
||||
def subscribe(
|
||||
self, job_id: str, *, after_seq: int | None = None
|
||||
) -> Subscription | None:
|
||||
"""SSE subscribe: get replay buffer + live events stream."""
|
||||
with self._lock:
|
||||
if self._job is None or self._job.job_id != job_id:
|
||||
return None
|
||||
q: queue.Queue = queue.Queue(maxsize = 2000)
|
||||
self._subs.append(q)
|
||||
if after_seq is None:
|
||||
replay = list(self._events)
|
||||
else:
|
||||
replay = [e for e in self._events if int(e.get("seq") or 0) > after_seq]
|
||||
return Subscription(replay = replay, _q = q)
|
||||
|
||||
def unsubscribe(self, sub: Subscription) -> None:
|
||||
"""Drop SSE subscriber (client disconnected)."""
|
||||
with self._lock:
|
||||
self._subs = [q for q in self._subs if q is not sub._q]
|
||||
|
||||
def _emit(self, event: dict) -> None:
|
||||
"""Broadcast event to replay buffer + all subscribers."""
|
||||
self._seq += 1
|
||||
event["seq"] = self._seq
|
||||
self._events.append(event)
|
||||
stale: list[queue.Queue] = []
|
||||
for q in self._subs:
|
||||
try:
|
||||
q.put_nowait(event)
|
||||
except queue.Full:
|
||||
stale.append(q)
|
||||
if stale:
|
||||
self._subs = [q for q in self._subs if q not in stale]
|
||||
|
||||
def _snapshot(self) -> tuple[Job, mp.Process, Any] | None:
|
||||
"""Grab pointers for the pump loop (avoid holding lock too long)."""
|
||||
with self._lock:
|
||||
if self._job is None or self._proc is None or self._mp_q is None:
|
||||
return None
|
||||
return self._job, self._proc, self._mp_q
|
||||
|
||||
@staticmethod
|
||||
def _read_queue_with_timeout(q: Any, *, timeout_sec: float) -> dict | None:
|
||||
"""Try read 1 event from mp queue. Timeout = pump stays responsive."""
|
||||
try:
|
||||
return coerce_event(q.get(timeout = timeout_sec))
|
||||
except queue.Empty:
|
||||
return None
|
||||
except (EOFError, OSError, ValueError):
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _drain_queue(q: Any) -> list[dict]:
|
||||
"""Drain mp queue fast (used on process exit)."""
|
||||
events: list[dict] = []
|
||||
while True:
|
||||
try:
|
||||
events.append(coerce_event(q.get_nowait()))
|
||||
except queue.Empty:
|
||||
return events
|
||||
except (EOFError, OSError, ValueError):
|
||||
return events
|
||||
|
||||
def _pump_loop(self) -> None:
|
||||
"""Background thread: consumes worker events + updates job snapshot."""
|
||||
while True:
|
||||
snap = self._snapshot()
|
||||
if snap is None:
|
||||
return
|
||||
job, proc, mp_q = snap
|
||||
|
||||
event = self._read_queue_with_timeout(mp_q, timeout_sec = 0.25)
|
||||
if event is not None:
|
||||
self._handle_event(job, event)
|
||||
continue
|
||||
|
||||
if proc.is_alive():
|
||||
continue
|
||||
|
||||
for e in self._drain_queue(mp_q):
|
||||
self._handle_event(job, e)
|
||||
|
||||
with self._lock:
|
||||
if self._job and self._job.status in {
|
||||
"pending",
|
||||
"active",
|
||||
"cancelling",
|
||||
}:
|
||||
if self._job.status == "cancelling":
|
||||
self._job.status = "cancelled"
|
||||
else:
|
||||
self._job.status = "error"
|
||||
self._job.error = self._job.error or "process exited"
|
||||
self._job.finished_at = time.time()
|
||||
event_type = (
|
||||
EVENT_JOB_CANCELLED
|
||||
if self._job.status == "cancelled"
|
||||
else EVENT_JOB_ERROR
|
||||
)
|
||||
self._emit(
|
||||
{
|
||||
"type": event_type,
|
||||
"ts": time.time(),
|
||||
"job_id": self._job.job_id,
|
||||
}
|
||||
)
|
||||
return
|
||||
|
||||
def _handle_event(self, job: Job, event: dict) -> None:
|
||||
"""Apply event -> job state + forward to SSE."""
|
||||
et = event.get("type")
|
||||
msg = event.get("message") if et == "log" else None
|
||||
|
||||
with self._lock:
|
||||
if self._job is None or self._job.job_id != job.job_id:
|
||||
return
|
||||
if et == EVENT_JOB_STARTED:
|
||||
self._job.status = "active"
|
||||
if et == EVENT_JOB_COMPLETED:
|
||||
self._job.status = "completed"
|
||||
self._job.finished_at = time.time()
|
||||
self._job.analysis = event.get("analysis")
|
||||
self._job.artifact_path = event.get("artifact_path")
|
||||
self._job.dataset = event.get("dataset")
|
||||
self._job.processor_artifacts = event.get("processor_artifacts")
|
||||
if self._job.progress.total and self._job.progress.total > 0:
|
||||
self._job.progress.done = self._job.progress.total
|
||||
self._job.progress.percent = 100.0
|
||||
if et == EVENT_JOB_ERROR:
|
||||
self._job.status = "error"
|
||||
self._job.finished_at = time.time()
|
||||
self._job.error = event.get("error") or "error"
|
||||
|
||||
if msg:
|
||||
upd = parse_log_message(msg)
|
||||
if upd:
|
||||
apply_update(self._job, upd)
|
||||
|
||||
self._emit(event)
|
||||
|
||||
|
||||
_JOB_MANAGER: JobManager | None = None
|
||||
|
||||
|
||||
def get_job_manager() -> JobManager:
|
||||
"""Singleton JobManager (we only run 1 job anyway)."""
|
||||
global _JOB_MANAGER
|
||||
if _JOB_MANAGER is None:
|
||||
_JOB_MANAGER = JobManager()
|
||||
return _JOB_MANAGER
|
||||
262
studio/backend/core/data_recipe/jobs/parse.py
Normal file
262
studio/backend/core/data_recipe/jobs/parse.py
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from .constants import (
|
||||
STAGE_BATCH,
|
||||
STAGE_COLUMN_CONFIG,
|
||||
STAGE_CREATE,
|
||||
STAGE_DAG,
|
||||
STAGE_GENERATING,
|
||||
STAGE_HEALTHCHECK,
|
||||
STAGE_PREVIEW,
|
||||
STAGE_PROFILING,
|
||||
STAGE_SAMPLING,
|
||||
USAGE_RESET_STAGES,
|
||||
)
|
||||
from .types import Job, ModelUsage, Progress
|
||||
|
||||
|
||||
@dataclass(frozen = True)
|
||||
class ParsedUpdate:
|
||||
stage: str | None = None
|
||||
current_column: str | None = None
|
||||
progress: Progress | None = None
|
||||
rows: int | None = None
|
||||
cols: int | None = None
|
||||
batch_idx: int | None = None
|
||||
batch_total: int | None = None
|
||||
usage_model: str | None = None
|
||||
usage_input_tokens: int | None = None
|
||||
usage_output_tokens: int | None = None
|
||||
usage_total_tokens: int | None = None
|
||||
usage_tps: float | None = None
|
||||
usage_requests_success: int | None = None
|
||||
usage_requests_failed: int | None = None
|
||||
usage_requests_total: int | None = None
|
||||
usage_rpm: float | None = None
|
||||
usage_section_start: bool | None = None
|
||||
|
||||
|
||||
# kinda of a bummber but currently only option, Best effort parser from data-designer logs -> structured status for UI.
|
||||
_RE_SAMPLERS = re.compile(
|
||||
r"Preparing samplers to generate (?P<rows>\d+) records across (?P<cols>\d+) columns"
|
||||
)
|
||||
_RE_COLCFG = re.compile(r"model config for column '(?P<col>[^']+)'")
|
||||
_RE_PROCESSING_COL = re.compile(r"Processing .* column '(?P<col>[^']+)'")
|
||||
_RE_PROGRESS = re.compile(
|
||||
r"progress: (?P<done>\d+)/(?P<total>\d+) \((?P<pct>\d+)%\) complete, "
|
||||
r"(?P<ok>\d+) ok, (?P<failed>\d+) failed, (?P<rate>[0-9.]+) rec/s, eta (?P<eta>[0-9.]+)s"
|
||||
)
|
||||
_RE_BATCH = re.compile(r"Processing batch (?P<idx>\d+) of (?P<total>\d+)")
|
||||
_RE_USAGE_MODEL = re.compile(r"model:\s*(?P<model>.+)$")
|
||||
_RE_USAGE_TOKENS = re.compile(
|
||||
r"tokens:\s*input=(?P<input>\d+),\s*output=(?P<output>\d+),\s*total=(?P<total>\d+),\s*tps=(?P<tps>[0-9.]+)"
|
||||
)
|
||||
_RE_USAGE_REQUESTS = re.compile(
|
||||
r"requests:\s*success=(?P<success>\d+),\s*failed=(?P<failed>\d+),\s*total=(?P<total>\d+),\s*rpm=(?P<rpm>[0-9.]+)"
|
||||
)
|
||||
|
||||
|
||||
def parse_log_message(msg: str) -> ParsedUpdate | None:
|
||||
m = _RE_SAMPLERS.search(msg)
|
||||
if m:
|
||||
return ParsedUpdate(
|
||||
stage = STAGE_SAMPLING,
|
||||
rows = int(m.group("rows")),
|
||||
cols = int(m.group("cols")),
|
||||
)
|
||||
|
||||
if "Sorting column configs into a Directed Acyclic Graph" in msg:
|
||||
return ParsedUpdate(stage = STAGE_DAG)
|
||||
if "Running health checks for models" in msg:
|
||||
return ParsedUpdate(stage = STAGE_HEALTHCHECK)
|
||||
if "Preview generation in progress" in msg:
|
||||
return ParsedUpdate(stage = STAGE_PREVIEW)
|
||||
if "Creating Data Designer dataset" in msg:
|
||||
return ParsedUpdate(stage = STAGE_CREATE)
|
||||
if "Measuring dataset column statistics" in msg:
|
||||
return ParsedUpdate(stage = STAGE_PROFILING)
|
||||
|
||||
m = _RE_COLCFG.search(msg)
|
||||
if m:
|
||||
col = m.group("col")
|
||||
return ParsedUpdate(stage = STAGE_COLUMN_CONFIG, current_column = col)
|
||||
|
||||
m = _RE_PROCESSING_COL.search(msg)
|
||||
if m:
|
||||
col = m.group("col")
|
||||
return ParsedUpdate(stage = STAGE_GENERATING, current_column = col)
|
||||
|
||||
m = _RE_PROGRESS.search(msg)
|
||||
if m:
|
||||
p = Progress(
|
||||
done = int(m.group("done")),
|
||||
total = int(m.group("total")),
|
||||
percent = float(m.group("pct")),
|
||||
ok = int(m.group("ok")),
|
||||
failed = int(m.group("failed")),
|
||||
rate = float(m.group("rate")),
|
||||
eta_sec = float(m.group("eta")),
|
||||
)
|
||||
return ParsedUpdate(stage = STAGE_GENERATING, progress = p)
|
||||
|
||||
m = _RE_BATCH.search(msg)
|
||||
if m:
|
||||
return ParsedUpdate(
|
||||
stage = STAGE_BATCH,
|
||||
batch_idx = int(m.group("idx")),
|
||||
batch_total = int(m.group("total")),
|
||||
)
|
||||
|
||||
if "Model usage summary" in msg:
|
||||
return ParsedUpdate(usage_section_start = True)
|
||||
|
||||
m = _RE_USAGE_MODEL.search(msg)
|
||||
if m and "|-- model:" in msg:
|
||||
return ParsedUpdate(usage_model = str(m.group("model")).strip())
|
||||
|
||||
m = _RE_USAGE_TOKENS.search(msg)
|
||||
if m:
|
||||
return ParsedUpdate(
|
||||
usage_input_tokens = int(m.group("input")),
|
||||
usage_output_tokens = int(m.group("output")),
|
||||
usage_total_tokens = int(m.group("total")),
|
||||
usage_tps = float(m.group("tps")),
|
||||
)
|
||||
|
||||
m = _RE_USAGE_REQUESTS.search(msg)
|
||||
if m:
|
||||
return ParsedUpdate(
|
||||
usage_requests_success = int(m.group("success")),
|
||||
usage_requests_failed = int(m.group("failed")),
|
||||
usage_requests_total = int(m.group("total")),
|
||||
usage_rpm = float(m.group("rpm")),
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def apply_update(job: Job, update: ParsedUpdate) -> None:
|
||||
if update.stage is not None:
|
||||
job.stage = update.stage
|
||||
if update.current_column is not None:
|
||||
job.current_column = update.current_column
|
||||
if (
|
||||
update.stage == STAGE_GENERATING
|
||||
and update.current_column not in job._seen_generation_columns
|
||||
):
|
||||
job._seen_generation_columns.append(update.current_column)
|
||||
if update.rows is not None:
|
||||
job.rows = update.rows
|
||||
if update.cols is not None:
|
||||
job.cols = update.cols
|
||||
if update.progress is not None:
|
||||
job.column_progress = update.progress
|
||||
if (
|
||||
job.current_column
|
||||
and update.progress.done is not None
|
||||
and update.progress.total is not None
|
||||
and update.progress.total > 0
|
||||
and update.progress.done >= update.progress.total
|
||||
and job.current_column not in job.completed_columns
|
||||
):
|
||||
job.completed_columns.append(job.current_column)
|
||||
job.progress = _compute_overall_progress(job, update.progress)
|
||||
if update.batch_idx is not None:
|
||||
job.batch.idx = update.batch_idx
|
||||
if update.batch_total is not None:
|
||||
job.batch.total = update.batch_total
|
||||
|
||||
if update.stage in USAGE_RESET_STAGES:
|
||||
# usage summary is a short block so we reset once we move into the next stage.
|
||||
job._in_usage_summary = False
|
||||
|
||||
if update.usage_section_start is not None:
|
||||
job._in_usage_summary = update.usage_section_start
|
||||
if update.usage_section_start:
|
||||
job._current_usage_model = None
|
||||
|
||||
if not job._in_usage_summary:
|
||||
return
|
||||
|
||||
if update.usage_model is not None:
|
||||
name = update.usage_model.strip().strip("'").strip('"')
|
||||
job._current_usage_model = name
|
||||
if name not in job.model_usage:
|
||||
job.model_usage[name] = ModelUsage(model = name)
|
||||
|
||||
if job._current_usage_model is None:
|
||||
return
|
||||
|
||||
usage = job.model_usage.get(job._current_usage_model)
|
||||
if usage is None:
|
||||
return
|
||||
|
||||
if update.usage_input_tokens is not None:
|
||||
usage.input_tokens = update.usage_input_tokens
|
||||
if update.usage_output_tokens is not None:
|
||||
usage.output_tokens = update.usage_output_tokens
|
||||
if update.usage_total_tokens is not None:
|
||||
usage.total_tokens = update.usage_total_tokens
|
||||
if update.usage_tps is not None:
|
||||
usage.tps = update.usage_tps
|
||||
if update.usage_requests_success is not None:
|
||||
usage.requests_success = update.usage_requests_success
|
||||
if update.usage_requests_failed is not None:
|
||||
usage.requests_failed = update.usage_requests_failed
|
||||
if update.usage_requests_total is not None:
|
||||
usage.requests_total = update.usage_requests_total
|
||||
if update.usage_rpm is not None:
|
||||
usage.rpm = update.usage_rpm
|
||||
|
||||
|
||||
def _compute_overall_progress(job: Job, column_progress: Progress) -> Progress:
|
||||
if not job.rows:
|
||||
return column_progress
|
||||
|
||||
total_rows = max(1, int(job.rows))
|
||||
current_done = 0 if column_progress.done is None else int(column_progress.done)
|
||||
current_done = max(0, min(current_done, total_rows))
|
||||
total_columns = max(1, int(job.progress_columns_total or 1))
|
||||
|
||||
if job.current_column:
|
||||
job._column_done[job.current_column] = current_done
|
||||
|
||||
if len(job._column_done) == 0:
|
||||
done = current_done
|
||||
else:
|
||||
sum_done = sum(
|
||||
max(0, min(value, total_rows)) for value in job._column_done.values()
|
||||
)
|
||||
done = int(sum_done / total_columns)
|
||||
|
||||
prev_done = int(job.progress.done or 0)
|
||||
if done < prev_done:
|
||||
done = prev_done
|
||||
if done > total_rows:
|
||||
done = total_rows
|
||||
percent = (done / total_rows) * 100 if total_rows > 0 else 100.0
|
||||
prev_percent = float(job.progress.percent or 0.0)
|
||||
if percent < prev_percent:
|
||||
percent = prev_percent
|
||||
|
||||
return Progress(
|
||||
done = done,
|
||||
total = total_rows,
|
||||
percent = percent,
|
||||
eta_sec = column_progress.eta_sec,
|
||||
rate = column_progress.rate,
|
||||
ok = column_progress.ok,
|
||||
failed = column_progress.failed,
|
||||
)
|
||||
|
||||
|
||||
def coerce_event(obj: Any) -> dict:
|
||||
"""Normalize worker payload into event dict."""
|
||||
return obj if isinstance(obj, dict) else {"type": "log", "message": str(obj)}
|
||||
76
studio/backend/core/data_recipe/jobs/types.py
Normal file
76
studio/backend/core/data_recipe/jobs/types.py
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal
|
||||
|
||||
|
||||
JobStatus = Literal[
|
||||
"created",
|
||||
"pending",
|
||||
"active",
|
||||
"cancelling",
|
||||
"cancelled",
|
||||
"error",
|
||||
"completed",
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Progress:
|
||||
done: int | None = None
|
||||
total: int | None = None
|
||||
percent: float | None = None
|
||||
eta_sec: float | None = None
|
||||
rate: float | None = None
|
||||
ok: int | None = None
|
||||
failed: int | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class BatchProgress:
|
||||
idx: int | None = None
|
||||
total: int | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ModelUsage:
|
||||
model: str
|
||||
input_tokens: int | None = None
|
||||
output_tokens: int | None = None
|
||||
total_tokens: int | None = None
|
||||
tps: float | None = None
|
||||
requests_success: int | None = None
|
||||
requests_failed: int | None = None
|
||||
requests_total: int | None = None
|
||||
rpm: float | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Job:
|
||||
job_id: str
|
||||
status: JobStatus = "created"
|
||||
stage: str | None = None
|
||||
current_column: str | None = None
|
||||
progress: Progress = field(default_factory = Progress)
|
||||
column_progress: Progress = field(default_factory = Progress)
|
||||
batch: BatchProgress = field(default_factory = BatchProgress)
|
||||
rows: int | None = None
|
||||
cols: int | None = None
|
||||
error: str | None = None
|
||||
started_at: float | None = None
|
||||
finished_at: float | None = None
|
||||
|
||||
analysis: dict[str, Any] | None = None
|
||||
artifact_path: str | None = None
|
||||
dataset: list[dict[str, Any]] | None = None
|
||||
processor_artifacts: dict[str, Any] | None = None
|
||||
model_usage: dict[str, ModelUsage] = field(default_factory = dict)
|
||||
progress_columns_total: int | None = None
|
||||
completed_columns: list[str] = field(default_factory = list)
|
||||
_current_usage_model: str | None = None
|
||||
_in_usage_summary: bool = False
|
||||
_seen_generation_columns: list[str] = field(default_factory = list)
|
||||
_column_done: dict[str, int] = field(default_factory = dict)
|
||||
239
studio/backend/core/data_recipe/jobs/worker.py
Normal file
239
studio/backend/core/data_recipe/jobs/worker.py
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import structlog
|
||||
import loggers
|
||||
import logging
|
||||
import re
|
||||
import shutil
|
||||
import time
|
||||
import traceback
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ..jsonable import to_jsonable, to_preview_jsonable
|
||||
from .constants import EVENT_JOB_COMPLETED, EVENT_JOB_ERROR, EVENT_JOB_STARTED
|
||||
from ..service import build_config_builder, create_data_designer
|
||||
from utils.paths import ensure_dir, recipe_datasets_root
|
||||
|
||||
_ARTIFACT_ROOT = recipe_datasets_root()
|
||||
|
||||
|
||||
class _QueueLogHandler(logging.Handler):
|
||||
def __init__(self, event_queue):
|
||||
super().__init__()
|
||||
self._q = event_queue
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
try:
|
||||
event = {
|
||||
"type": "log",
|
||||
"ts": record.created,
|
||||
"level": record.levelname,
|
||||
"logger": record.name,
|
||||
"message": record.getMessage(),
|
||||
}
|
||||
self._q.put(event)
|
||||
except (OSError, RuntimeError, ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _slugify_run_name(value: str) -> str:
|
||||
normalized = unicodedata.normalize("NFKD", value)
|
||||
ascii_only = normalized.encode("ascii", "ignore").decode("ascii")
|
||||
slug = re.sub(r"[^a-zA-Z0-9]+", "-", ascii_only).strip("-").lower()
|
||||
if not slug:
|
||||
return ""
|
||||
return slug[:80].strip("-")
|
||||
|
||||
|
||||
def _build_dataset_name(
|
||||
*, run_name: str | None, job_id: str, artifact_root: Path
|
||||
) -> str:
|
||||
fallback = f"recipe_{job_id}"
|
||||
slug = _slugify_run_name(run_name or "")
|
||||
base_name = f"recipe_{slug}" if slug else fallback
|
||||
candidate = base_name
|
||||
suffix = 2
|
||||
while (artifact_root / candidate).exists():
|
||||
candidate = f"{base_name}_{suffix}"
|
||||
suffix += 1
|
||||
return candidate
|
||||
|
||||
|
||||
def run_job_process(
|
||||
*,
|
||||
event_queue,
|
||||
recipe: dict[str, Any],
|
||||
run: dict[str, Any],
|
||||
) -> None:
|
||||
"""
|
||||
Subprocess entrypoint.
|
||||
Sends events to `event_queue`.
|
||||
"""
|
||||
import os
|
||||
|
||||
os.environ["PYTHONWARNINGS"] = (
|
||||
"ignore" # Suppress warnings at C-level before imports
|
||||
)
|
||||
|
||||
import warnings
|
||||
from loggers.config import LogConfig
|
||||
|
||||
if os.getenv("ENVIRONMENT_TYPE", "production") == "production":
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
LogConfig.setup_logging(
|
||||
service_name = "unsloth-studio-data-worker",
|
||||
env = os.getenv("ENVIRONMENT_TYPE", "production"),
|
||||
)
|
||||
|
||||
event_queue.put({"type": EVENT_JOB_STARTED, "ts": time.time()})
|
||||
|
||||
try:
|
||||
from data_designer.config.run_config import RunConfig
|
||||
|
||||
rows = int(run.get("rows") or 1000)
|
||||
job_id = str(run.get("_job_id") or "").strip()
|
||||
if not job_id:
|
||||
job_id = f"{int(time.time())}"
|
||||
run_name_raw = run.get("run_name")
|
||||
run_name = run_name_raw if isinstance(run_name_raw, str) else None
|
||||
dataset_name = _build_dataset_name(
|
||||
run_name = run_name,
|
||||
job_id = job_id,
|
||||
artifact_root = _ARTIFACT_ROOT,
|
||||
)
|
||||
merge_batches = bool(run.get("merge_batches"))
|
||||
ensure_dir(_ARTIFACT_ROOT)
|
||||
run_config_raw = run.get("run_config") or {}
|
||||
|
||||
builder = build_config_builder(recipe)
|
||||
designer = create_data_designer(recipe, artifact_path = str(_ARTIFACT_ROOT))
|
||||
|
||||
# DataDesigner configures root logging in DataDesigner.__init__.
|
||||
# Attach queue logger directly to `data_designer` so parser events survive root resets.
|
||||
handler = _QueueLogHandler(event_queue)
|
||||
handler.setLevel(logging.INFO)
|
||||
data_designer_logger = logging.getLogger("data_designer")
|
||||
data_designer_logger.addHandler(handler)
|
||||
data_designer_logger.setLevel(logging.INFO)
|
||||
data_designer_logger.propagate = True
|
||||
|
||||
if run_config_raw:
|
||||
designer.set_run_config(RunConfig.model_validate(run_config_raw))
|
||||
|
||||
execution_type = str(run.get("execution_type") or "full").strip().lower()
|
||||
if execution_type == "preview":
|
||||
results = designer.preview(builder, num_records = rows)
|
||||
analysis = (
|
||||
None
|
||||
if results.analysis is None
|
||||
else to_jsonable(results.analysis.model_dump(mode = "json"))
|
||||
)
|
||||
dataset = (
|
||||
[]
|
||||
if results.dataset is None
|
||||
else to_preview_jsonable(results.dataset.to_dict(orient = "records"))
|
||||
)
|
||||
processor_artifacts = (
|
||||
None
|
||||
if results.processor_artifacts is None
|
||||
else to_jsonable(results.processor_artifacts)
|
||||
)
|
||||
event_queue.put(
|
||||
{
|
||||
"type": EVENT_JOB_COMPLETED,
|
||||
"ts": time.time(),
|
||||
"analysis": analysis,
|
||||
"dataset": dataset,
|
||||
"processor_artifacts": processor_artifacts,
|
||||
"artifact_path": None,
|
||||
"execution_type": execution_type,
|
||||
}
|
||||
)
|
||||
else:
|
||||
results = designer.create(
|
||||
builder, num_records = rows, dataset_name = dataset_name
|
||||
)
|
||||
analysis = to_jsonable(results.load_analysis().model_dump(mode = "json"))
|
||||
if merge_batches:
|
||||
_merge_batches_to_single_parquet(
|
||||
results.artifact_storage.base_dataset_path
|
||||
)
|
||||
artifact_path = str(results.artifact_storage.base_dataset_path)
|
||||
event_queue.put(
|
||||
{
|
||||
"type": EVENT_JOB_COMPLETED,
|
||||
"ts": time.time(),
|
||||
"analysis": analysis,
|
||||
"artifact_path": artifact_path,
|
||||
"execution_type": execution_type,
|
||||
}
|
||||
)
|
||||
except Exception as exc:
|
||||
event_queue.put(
|
||||
{
|
||||
"type": EVENT_JOB_ERROR,
|
||||
"ts": time.time(),
|
||||
"error": str(exc),
|
||||
"stack": traceback.format_exc(limit = 20),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _merge_batches_to_single_parquet(base_dataset_path: Path) -> None:
|
||||
parquet_dir = base_dataset_path / "parquet-files"
|
||||
parquet_files = sorted(parquet_dir.glob("*.parquet"))
|
||||
if len(parquet_files) <= 1:
|
||||
return
|
||||
|
||||
try:
|
||||
from data_designer.config.utils.io_helpers import read_parquet_dataset
|
||||
except ImportError:
|
||||
return
|
||||
|
||||
dataframe = read_parquet_dataset(parquet_dir)
|
||||
shutil.rmtree(parquet_dir)
|
||||
parquet_dir.mkdir(parents = True, exist_ok = True)
|
||||
merged_file = parquet_dir / "batch_00000.parquet"
|
||||
dataframe.to_parquet(merged_file, index = False)
|
||||
_rewrite_merged_metadata(
|
||||
base_dataset_path = base_dataset_path,
|
||||
parquet_file = merged_file,
|
||||
)
|
||||
|
||||
|
||||
def _rewrite_merged_metadata(*, base_dataset_path: Path, parquet_file: Path) -> None:
|
||||
metadata_path = base_dataset_path / "metadata.json"
|
||||
if not metadata_path.exists():
|
||||
return
|
||||
|
||||
try:
|
||||
metadata = json.loads(metadata_path.read_text(encoding = "utf-8"))
|
||||
except (OSError, TypeError, ValueError):
|
||||
return
|
||||
|
||||
if not isinstance(metadata, dict):
|
||||
return
|
||||
|
||||
relative_parquet_path = str(parquet_file.relative_to(base_dataset_path))
|
||||
file_paths = metadata.get("file_paths")
|
||||
if not isinstance(file_paths, dict):
|
||||
file_paths = {}
|
||||
file_paths["parquet-files"] = [relative_parquet_path]
|
||||
metadata["file_paths"] = file_paths
|
||||
metadata["total_num_batches"] = 1
|
||||
metadata["num_completed_batches"] = 1
|
||||
|
||||
try:
|
||||
metadata_path.write_text(
|
||||
json.dumps(metadata, indent = 2, sort_keys = True),
|
||||
encoding = "utf-8",
|
||||
)
|
||||
except OSError:
|
||||
return
|
||||
121
studio/backend/core/data_recipe/jsonable.py
Normal file
121
studio/backend/core/data_recipe/jsonable.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import io
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _pil_to_preview_payload(image: Any) -> dict[str, Any]:
|
||||
buffer = io.BytesIO()
|
||||
image.convert("RGB").save(buffer, format = "JPEG", quality = 85)
|
||||
return {
|
||||
"type": "image",
|
||||
"mime": "image/jpeg",
|
||||
"width": image.width,
|
||||
"height": image.height,
|
||||
"data": base64.b64encode(buffer.getvalue()).decode("ascii"),
|
||||
}
|
||||
|
||||
|
||||
def _open_pil_image_from_bytes(raw_bytes: bytes):
|
||||
from PIL import Image # type: ignore
|
||||
|
||||
with Image.open(io.BytesIO(raw_bytes)) as image:
|
||||
return image.copy()
|
||||
|
||||
|
||||
def _to_pil_from_hf_image_dict(value: Any) -> Any | None:
|
||||
if not isinstance(value, dict):
|
||||
return None
|
||||
|
||||
raw_bytes = value.get("bytes")
|
||||
if isinstance(raw_bytes, (bytes, bytearray)) and len(raw_bytes) > 0:
|
||||
try:
|
||||
return _open_pil_image_from_bytes(bytes(raw_bytes))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
if (
|
||||
isinstance(raw_bytes, list)
|
||||
and len(raw_bytes) > 0
|
||||
and all(isinstance(item, int) and 0 <= item <= 255 for item in raw_bytes)
|
||||
):
|
||||
try:
|
||||
return _open_pil_image_from_bytes(bytes(raw_bytes))
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
path_value = value.get("path")
|
||||
if isinstance(path_value, str) and path_value.strip():
|
||||
try:
|
||||
from PIL import Image # type: ignore
|
||||
|
||||
with Image.open(Path(path_value)) as image:
|
||||
return image.copy()
|
||||
except (OSError, ValueError, TypeError):
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def to_jsonable(value: Any) -> Any:
|
||||
"""Convert numpy/pandas-ish values into plain JSON-safe values."""
|
||||
try:
|
||||
import numpy as np # type: ignore
|
||||
except ImportError: # pragma: no cover
|
||||
np = None # type: ignore
|
||||
|
||||
if np is not None:
|
||||
if isinstance(value, np.ndarray):
|
||||
return value.tolist()
|
||||
if isinstance(value, np.generic):
|
||||
return value.item()
|
||||
|
||||
if isinstance(value, dict):
|
||||
return {str(k): to_jsonable(v) for k, v in value.items()}
|
||||
if isinstance(value, (list, tuple, set)):
|
||||
return [to_jsonable(v) for v in value]
|
||||
|
||||
if hasattr(value, "isoformat") and callable(value.isoformat):
|
||||
try:
|
||||
return value.isoformat()
|
||||
except (TypeError, ValueError):
|
||||
return value
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def _to_preview_image_payload(value: Any) -> dict[str, Any] | None:
|
||||
try:
|
||||
from PIL.Image import Image as PILImage # type: ignore
|
||||
except ImportError: # pragma: no cover
|
||||
return None
|
||||
|
||||
if not isinstance(value, PILImage):
|
||||
hf_image = _to_pil_from_hf_image_dict(value)
|
||||
if hf_image is None:
|
||||
return None
|
||||
value = hf_image
|
||||
|
||||
return _pil_to_preview_payload(value)
|
||||
|
||||
|
||||
def to_preview_jsonable(value: Any) -> Any:
|
||||
"""Convert values into JSON-safe preview values, including PIL images."""
|
||||
image_payload = _to_preview_image_payload(value)
|
||||
if image_payload is not None:
|
||||
return image_payload
|
||||
|
||||
converted = to_jsonable(value)
|
||||
if converted is None or isinstance(converted, (str, int, float, bool)):
|
||||
return converted
|
||||
if isinstance(converted, dict):
|
||||
return {str(k): to_preview_jsonable(v) for k, v in converted.items()}
|
||||
if isinstance(converted, (list, tuple, set)):
|
||||
return [to_preview_jsonable(v) for v in converted]
|
||||
if isinstance(converted, (bytes, bytearray)):
|
||||
return base64.b64encode(bytes(converted)).decode("ascii")
|
||||
return str(converted)
|
||||
340
studio/backend/core/data_recipe/local_callable_validators.py
Normal file
340
studio/backend/core/data_recipe/local_callable_validators.py
Normal file
|
|
@ -0,0 +1,340 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import structlog
|
||||
import subprocess
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from loggers import get_logger
|
||||
from utils.paths import ensure_dir, oxc_validator_tmp_root
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
OXC_VALIDATION_FN_MARKER = "unsloth_oxc_validator"
|
||||
|
||||
_OXC_LANG_TO_NODE_LANG = {
|
||||
"javascript": "js",
|
||||
"typescript": "ts",
|
||||
"jsx": "jsx",
|
||||
"tsx": "tsx",
|
||||
}
|
||||
_OXC_VALIDATION_MODES = {"syntax", "lint", "syntax+lint"}
|
||||
_OXC_CODE_SHAPES = {"auto", "module", "snippet"}
|
||||
|
||||
_OXC_TOOL_DIR = Path(__file__).resolve().parent / "oxc-validator"
|
||||
_OXC_RUNNER_PATH = _OXC_TOOL_DIR / "validate.mjs"
|
||||
|
||||
|
||||
@dataclass(frozen = True)
|
||||
class OxcLocalCallableValidatorSpec:
|
||||
name: str
|
||||
drop: bool
|
||||
target_columns: list[str]
|
||||
batch_size: int
|
||||
code_lang: str
|
||||
validation_mode: str
|
||||
code_shape: str
|
||||
|
||||
|
||||
def split_oxc_local_callable_validators(
|
||||
recipe_core: dict[str, Any],
|
||||
) -> tuple[dict[str, Any], list[OxcLocalCallableValidatorSpec]]:
|
||||
columns = recipe_core.get("columns")
|
||||
if not isinstance(columns, list):
|
||||
return recipe_core, []
|
||||
|
||||
sanitized = deepcopy(recipe_core)
|
||||
sanitized_columns = sanitized.get("columns")
|
||||
if not isinstance(sanitized_columns, list):
|
||||
return sanitized, []
|
||||
|
||||
kept_columns: list[Any] = []
|
||||
oxc_specs: list[OxcLocalCallableValidatorSpec] = []
|
||||
|
||||
for column in sanitized_columns:
|
||||
if not isinstance(column, dict):
|
||||
kept_columns.append(column)
|
||||
continue
|
||||
|
||||
maybe_spec = _parse_oxc_spec(column = column)
|
||||
if maybe_spec is None:
|
||||
kept_columns.append(column)
|
||||
continue
|
||||
oxc_specs.append(maybe_spec)
|
||||
|
||||
sanitized["columns"] = kept_columns
|
||||
return sanitized, oxc_specs
|
||||
|
||||
|
||||
def register_oxc_local_callable_validators(
|
||||
*,
|
||||
builder,
|
||||
specs: list[OxcLocalCallableValidatorSpec],
|
||||
) -> None:
|
||||
if not specs:
|
||||
return
|
||||
|
||||
from data_designer.config.column_configs import ValidationColumnConfig
|
||||
from data_designer.config.validator_params import (
|
||||
LocalCallableValidatorParams,
|
||||
ValidatorType,
|
||||
)
|
||||
|
||||
for spec in specs:
|
||||
validation_function = _build_oxc_validation_function(
|
||||
spec.code_lang,
|
||||
spec.validation_mode,
|
||||
spec.code_shape,
|
||||
)
|
||||
builder.add_column(
|
||||
ValidationColumnConfig(
|
||||
name = spec.name,
|
||||
drop = spec.drop,
|
||||
target_columns = spec.target_columns,
|
||||
validator_type = ValidatorType.LOCAL_CALLABLE,
|
||||
validator_params = LocalCallableValidatorParams(
|
||||
validation_function = validation_function,
|
||||
),
|
||||
batch_size = spec.batch_size,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _parse_oxc_spec(
|
||||
*,
|
||||
column: dict[str, Any],
|
||||
) -> OxcLocalCallableValidatorSpec | None:
|
||||
if str(column.get("column_type") or "").strip() != "validation":
|
||||
return None
|
||||
if str(column.get("validator_type") or "").strip() != "local_callable":
|
||||
return None
|
||||
|
||||
params = column.get("validator_params")
|
||||
if not isinstance(params, dict):
|
||||
return None
|
||||
|
||||
fn_raw = params.get("validation_function")
|
||||
fn_name = fn_raw.strip() if isinstance(fn_raw, str) else ""
|
||||
if not fn_name.startswith(OXC_VALIDATION_FN_MARKER):
|
||||
return None
|
||||
|
||||
name = str(column.get("name") or "").strip()
|
||||
if not name:
|
||||
return None
|
||||
|
||||
target_columns_raw = column.get("target_columns")
|
||||
target_columns = (
|
||||
[
|
||||
value.strip()
|
||||
for value in target_columns_raw
|
||||
if isinstance(value, str) and value.strip()
|
||||
]
|
||||
if isinstance(target_columns_raw, list)
|
||||
else []
|
||||
)
|
||||
if not target_columns:
|
||||
return None
|
||||
|
||||
code_lang, validation_mode, code_shape = _parse_oxc_validation_marker(fn_name)
|
||||
batch_size = _parse_batch_size(column.get("batch_size"))
|
||||
drop = bool(column.get("drop") is True)
|
||||
|
||||
return OxcLocalCallableValidatorSpec(
|
||||
name = name,
|
||||
drop = drop,
|
||||
target_columns = target_columns,
|
||||
batch_size = batch_size,
|
||||
code_lang = code_lang,
|
||||
validation_mode = validation_mode,
|
||||
code_shape = code_shape,
|
||||
)
|
||||
|
||||
|
||||
def _parse_batch_size(value: Any) -> int:
|
||||
try:
|
||||
parsed = int(value)
|
||||
except (TypeError, ValueError):
|
||||
return 10
|
||||
return parsed if parsed >= 1 else 10
|
||||
|
||||
|
||||
def _parse_oxc_validation_marker(fn_name: str) -> tuple[str, str, str]:
|
||||
marker = f"{OXC_VALIDATION_FN_MARKER}:"
|
||||
if not fn_name.startswith(marker):
|
||||
return "javascript", "syntax", "auto"
|
||||
suffix = fn_name[len(marker) :]
|
||||
parts = [part.strip() for part in suffix.split(":") if part.strip()]
|
||||
if len(parts) < 2:
|
||||
return "javascript", "syntax", "auto"
|
||||
code_lang = parts[0] if parts[0] in _OXC_LANG_TO_NODE_LANG else "javascript"
|
||||
mode = parts[1] if parts[1] in _OXC_VALIDATION_MODES else "syntax"
|
||||
code_shape = (
|
||||
parts[2] if len(parts) >= 3 and parts[2] in _OXC_CODE_SHAPES else "auto"
|
||||
)
|
||||
return code_lang, mode, code_shape
|
||||
|
||||
|
||||
@lru_cache(maxsize = 8)
|
||||
def _build_oxc_validation_function(lang: str, validation_mode: str, code_shape: str):
|
||||
node_lang = _OXC_LANG_TO_NODE_LANG.get(lang, "js")
|
||||
mode = validation_mode if validation_mode in _OXC_VALIDATION_MODES else "syntax"
|
||||
normalized_code_shape = code_shape if code_shape in _OXC_CODE_SHAPES else "auto"
|
||||
|
||||
def _validator(df):
|
||||
import pandas as pd # imported lazily for local callable runtime
|
||||
|
||||
row_count = int(len(df.index))
|
||||
if row_count == 0:
|
||||
return pd.DataFrame({"is_valid": []})
|
||||
|
||||
code_column = str(df.columns[0]) if len(df.columns) > 0 else ""
|
||||
code_values = (
|
||||
["" for _ in range(row_count)]
|
||||
if not code_column
|
||||
else [
|
||||
"" if value is None else str(value)
|
||||
for value in df[code_column].tolist()
|
||||
]
|
||||
)
|
||||
|
||||
results = _run_oxc_batch(
|
||||
node_lang = node_lang,
|
||||
validation_mode = mode,
|
||||
code_shape = normalized_code_shape,
|
||||
code_values = code_values,
|
||||
)
|
||||
if len(results) != row_count:
|
||||
results = _fallback_results(
|
||||
row_count,
|
||||
"OXC validator returned mismatched result size.",
|
||||
)
|
||||
return pd.DataFrame(results)
|
||||
|
||||
_validator.__name__ = f"{OXC_VALIDATION_FN_MARKER}_{node_lang}_{mode.replace('+', '_')}_{normalized_code_shape}"
|
||||
return _validator
|
||||
|
||||
|
||||
def _run_oxc_batch(
|
||||
*,
|
||||
node_lang: str,
|
||||
validation_mode: str,
|
||||
code_shape: str,
|
||||
code_values: list[str],
|
||||
) -> list[dict[str, Any]]:
|
||||
if not _OXC_RUNNER_PATH.exists():
|
||||
return _fallback_results(
|
||||
len(code_values),
|
||||
f"OXC runner missing at {_OXC_RUNNER_PATH}",
|
||||
)
|
||||
|
||||
payload = {
|
||||
"lang": node_lang,
|
||||
"mode": validation_mode,
|
||||
"code_shape": code_shape,
|
||||
"codes": code_values,
|
||||
}
|
||||
try:
|
||||
tmp_dir = ensure_dir(oxc_validator_tmp_root())
|
||||
env = dict(os.environ)
|
||||
tmp_dir_str = str(tmp_dir)
|
||||
env["TMPDIR"] = tmp_dir_str
|
||||
env["TMP"] = tmp_dir_str
|
||||
env["TEMP"] = tmp_dir_str
|
||||
proc = subprocess.run(
|
||||
["node", str(_OXC_RUNNER_PATH)],
|
||||
cwd = str(_OXC_TOOL_DIR),
|
||||
input = json.dumps(payload),
|
||||
text = True,
|
||||
capture_output = True,
|
||||
check = False,
|
||||
env = env,
|
||||
)
|
||||
except (OSError, ValueError) as exc:
|
||||
logger.warning("OXC subprocess launch failed: %s", exc)
|
||||
return _fallback_results(len(code_values), f"OXC launch failed: {exc}")
|
||||
|
||||
if proc.returncode != 0:
|
||||
message = (proc.stderr or proc.stdout or "unknown error").strip()
|
||||
if len(message) > 300:
|
||||
message = f"{message[:300]}..."
|
||||
return _fallback_results(len(code_values), f"OXC failed: {message}")
|
||||
|
||||
try:
|
||||
raw = json.loads(proc.stdout)
|
||||
except json.JSONDecodeError:
|
||||
return _fallback_results(len(code_values), "OXC output parse failed.")
|
||||
|
||||
if not isinstance(raw, list):
|
||||
return _fallback_results(len(code_values), "OXC output must be an array.")
|
||||
|
||||
out: list[dict[str, Any]] = []
|
||||
for item in raw:
|
||||
if not isinstance(item, dict):
|
||||
out.append(
|
||||
{
|
||||
"is_valid": False,
|
||||
"error_count": 1,
|
||||
"error_message": "Invalid OXC result entry.",
|
||||
"severity": None,
|
||||
"code": None,
|
||||
"labels": [],
|
||||
"codeframe": None,
|
||||
"warning_count": 0,
|
||||
}
|
||||
)
|
||||
continue
|
||||
is_valid_raw = item.get("is_valid")
|
||||
error_count_raw = item.get("error_count")
|
||||
message_raw = item.get("error_message")
|
||||
severity_raw = item.get("severity")
|
||||
code_raw = item.get("code")
|
||||
labels_raw = item.get("labels")
|
||||
codeframe_raw = item.get("codeframe")
|
||||
warning_count_raw = item.get("warning_count")
|
||||
out.append(
|
||||
{
|
||||
"is_valid": bool(is_valid_raw)
|
||||
if isinstance(is_valid_raw, bool)
|
||||
else False,
|
||||
"error_count": int(error_count_raw)
|
||||
if isinstance(error_count_raw, int)
|
||||
else 0,
|
||||
"error_message": str(message_raw or ""),
|
||||
"severity": str(severity_raw)
|
||||
if isinstance(severity_raw, str)
|
||||
else None,
|
||||
"code": str(code_raw) if isinstance(code_raw, str) else None,
|
||||
"labels": labels_raw if isinstance(labels_raw, list) else [],
|
||||
"codeframe": str(codeframe_raw)
|
||||
if isinstance(codeframe_raw, str)
|
||||
else None,
|
||||
"warning_count": int(warning_count_raw)
|
||||
if isinstance(warning_count_raw, int)
|
||||
else 0,
|
||||
}
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def _fallback_results(row_count: int, message: str) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{
|
||||
"is_valid": False,
|
||||
"error_count": 1,
|
||||
"error_message": message,
|
||||
"severity": None,
|
||||
"code": None,
|
||||
"labels": [],
|
||||
"codeframe": None,
|
||||
"warning_count": 0,
|
||||
}
|
||||
for _ in range(row_count)
|
||||
]
|
||||
794
studio/backend/core/data_recipe/oxc-validator/package-lock.json
generated
Normal file
794
studio/backend/core/data_recipe/oxc-validator/package-lock.json
generated
Normal file
|
|
@ -0,0 +1,794 @@
|
|||
{
|
||||
"name": "unsloth-oxc-validator-runtime",
|
||||
"version": "0.0.1",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "unsloth-oxc-validator-runtime",
|
||||
"version": "0.0.1",
|
||||
"dependencies": {
|
||||
"oxc-parser": "^0.116.0",
|
||||
"oxlint": "^1.51.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/core": {
|
||||
"version": "1.8.1",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.8.1.tgz",
|
||||
"integrity": "sha512-AvT9QFpxK0Zd8J0jopedNm+w/2fIzvtPKPjqyw9jwvBaReTTqPBk9Hixaz7KbjimP+QNz605/XnjFcDAL2pqBg==",
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@emnapi/wasi-threads": "1.1.0",
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/runtime": {
|
||||
"version": "1.8.1",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.8.1.tgz",
|
||||
"integrity": "sha512-mehfKSMWjjNol8659Z8KxEMrdSJDDot5SXMq00dM8BN4o+CLNXQ0xH2V7EchNHV4RmbZLmmPdEaXZc5H2FXmDg==",
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/wasi-threads": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.1.0.tgz",
|
||||
"integrity": "sha512-WI0DdZ8xFSbgMjR1sFsKABJ/C5OnRrjT06JXbZKexJGrDuPTzZdDYfFlsgcCXCyf+suG5QU2e/y1Wo2V/OapLQ==",
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/wasm-runtime": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.1.tgz",
|
||||
"integrity": "sha512-p64ah1M1ld8xjWv3qbvFwHiFVWrq1yFvV4f7w+mzaqiR4IlSgkqhcRdHwsGgomwzBH51sRY4NEowLxnaBjcW/A==",
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@emnapi/core": "^1.7.1",
|
||||
"@emnapi/runtime": "^1.7.1",
|
||||
"@tybys/wasm-util": "^0.10.1"
|
||||
},
|
||||
"funding": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/Brooooooklyn"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-android-arm-eabi": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm-eabi/-/binding-android-arm-eabi-0.116.0.tgz",
|
||||
"integrity": "sha512-AOET7YIOU3+ANO/3xQeRVGN5Xx6+JGXaIwlqkcHSfxJ/zzw2B6jb0YaLhX45SeRluKVTU8rka4N/tHtNoJjoCg==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-android-arm64": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-android-arm64/-/binding-android-arm64-0.116.0.tgz",
|
||||
"integrity": "sha512-yh0Zvth5cQ6XZkP3QF9MDrXf695zr5XxXq/wBQqpZb0uAgI9wpr98/Hx2RZITMfnNjkIq2VcyU44o3A0bdEmlQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-darwin-arm64": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-arm64/-/binding-darwin-arm64-0.116.0.tgz",
|
||||
"integrity": "sha512-plcTd/Jska55dToZz6XdRBPRVsj+asjD8QCpQFvt3Wj8pY+10D1pE53Mei3POAS/wSRSy7HiQ2twrm7H2A0CjA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-darwin-x64": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-darwin-x64/-/binding-darwin-x64-0.116.0.tgz",
|
||||
"integrity": "sha512-ahqcF3e3x5Z2ZepzXpZ8ugREdmxvBL+g1nQ0SxO11pIZfck6UtbOtwtdAAxnQXBHHtidu7lPcrBq1SEx26t1PQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-freebsd-x64": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-freebsd-x64/-/binding-freebsd-x64-0.116.0.tgz",
|
||||
"integrity": "sha512-yo2/LaSXtlzKBurvNbwun/sN/RJwW3XhbMr069FwNVtft7GBnaLLdPIz/sf47icxw/BPViEX6wFvzeD12mtrAg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-arm-gnueabihf": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.116.0.tgz",
|
||||
"integrity": "sha512-EiZeliIPPdFsuaPx8PzDMVijD/4YaUxO46/eYPk5raRocJqjjxOG6GAacQ8UrG3fbrgYjaEChfYL1e8DyE445A==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-arm-musleabihf": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.116.0.tgz",
|
||||
"integrity": "sha512-Nf7hnKRYRSIgglQcLAqE2St4b/Yr6dh+Z7in8mxol065Knevw71XZAiV1fmPSojq6uKPLV9eoH/wFrgr4TnZXw==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-arm64-gnu": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.116.0.tgz",
|
||||
"integrity": "sha512-9SJI0S4Qggn3QHpT8Y1jtZceA0m4BlpvO3ne2Wxd33UdTHMmelAnrXryjWutHWQtjCzOwSnFBEoQAdNNyt1u3A==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-arm64-musl": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.116.0.tgz",
|
||||
"integrity": "sha512-wMZ6//GI+q1JwO7G2OR51+eA5P8Gr3BobU8RAzCGJptvyGMkWb7KQ1E8s8naVZRr6bSGWAL2p3mCzKOxmEPmrA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-ppc64-gnu": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.116.0.tgz",
|
||||
"integrity": "sha512-5BO0KCzTG2HZTnp3r6SCAOeCs/GwFBQJ1WAOG/ROfDf1fVVEy6hrtLKTLCuUMaamH38v+1+RVEmzRkzBj+rMDQ==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-riscv64-gnu": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.116.0.tgz",
|
||||
"integrity": "sha512-M24gYb/ocVMnLwnH2wY5sLt4sRBkAUHDmfiYtyUYdKTkfPOKtpopd5otsL/BPLnIhpMD8zby4uXVvw7BU0UIlw==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-riscv64-musl": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.116.0.tgz",
|
||||
"integrity": "sha512-LHLXTHCH0bdvGjlitwr1ngeh32GAgq9HYzQ5VAgt0k0UT84AS8AkXj9Spoa9l20fXkVgSvAKcCEkydi4Ol23Dw==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-s390x-gnu": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.116.0.tgz",
|
||||
"integrity": "sha512-VE+XsztuE5jdHvLIDIQMuyDpz5NJGq1Vx/8EXYF0sS/gehlv9GhDpGVWU0SCZ/LjzIy4io/Z0W84UudqufvP3g==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-x64-gnu": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.116.0.tgz",
|
||||
"integrity": "sha512-rxUkauyjjCmgA7BoR63ogRGEtgubROnCm8AXE9ydg+p42jCGLLqG05mFcS2eC+FYyAU58ZFJNXXeqFW1iCyTGQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-linux-x64-musl": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-linux-x64-musl/-/binding-linux-x64-musl-0.116.0.tgz",
|
||||
"integrity": "sha512-0zoZlk9MmXe6oTgSh5lT1D51SDC1bfwC96JmE1amMFAPdEbJk5MFRisfTN9TFBpBigQua65842tjaxqMiorAYw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-openharmony-arm64": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-openharmony-arm64/-/binding-openharmony-arm64-0.116.0.tgz",
|
||||
"integrity": "sha512-PGS7Xqik77U9WMyW626gAD5A2rSN629UvyYJKAl/tgpT+KqZI4+56pJfExhv8IW/PpSHjYHwjmakwobLikz8ww==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openharmony"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-wasm32-wasi": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-wasm32-wasi/-/binding-wasm32-wasi-0.116.0.tgz",
|
||||
"integrity": "sha512-lGNf/9PU8XxB4Gt1Gr1AKwSrjxGYa6os0PlrT4bpoQsfE3gaZonQTKwJyKhiQdgy7pBCI+ed1LB1NNib1FYULw==",
|
||||
"cpu": [
|
||||
"wasm32"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@napi-rs/wasm-runtime": "^1.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=14.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-win32-arm64-msvc": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.116.0.tgz",
|
||||
"integrity": "sha512-tcsOHE31duBSRQXZ7NfdtjmMKZwQYlS00PwAMJ4w5oXs3iPCvisUuIXP7Ko4FzeOBTRvkd64btxtQ6cRM0Kwlw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-win32-ia32-msvc": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.116.0.tgz",
|
||||
"integrity": "sha512-higCz/x+dOQ264YEk22hnu4RDqvjhfehjFORpxoh42QyUxsP6eIembYesBUu5ilALWo0HvRD+m89az2BSTwqpQ==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-parser/binding-win32-x64-msvc": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-parser/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.116.0.tgz",
|
||||
"integrity": "sha512-Lg2SRmVHpGG85knDVLbv44r1bYn0OpIV0vg9jVmoEIpDj3Q4kwXuQ6MWVtuslwHR8o2CSiqdBeEn1n1URrs6Eg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-project/types": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.116.0.tgz",
|
||||
"integrity": "sha512-uOT8S1tlPmDckNxMNtIudN/yXpLdnhlJMX2oLS7cxCd7L0sUF09A/EbSVMWT3Y/iT44IwXCJSJfgfSxXAqWf9Q==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/Boshen"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-android-arm-eabi": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-android-arm-eabi/-/binding-android-arm-eabi-1.51.0.tgz",
|
||||
"integrity": "sha512-jJYIqbx4sX+suIxWstc4P7SzhEwb4ArWA2KVrmEuu9vH2i0qM6QIHz/ehmbGE4/2fZbpuMuBzTl7UkfNoqiSgw==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-android-arm64": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-android-arm64/-/binding-android-arm64-1.51.0.tgz",
|
||||
"integrity": "sha512-GtXyBCcH4ti98YdiMNCrpBNGitx87EjEWxevnyhcBK12k/Vu4EzSB45rzSC4fGFUD6sQgeaxItRCEEWeVwPafw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"android"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-darwin-arm64": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-darwin-arm64/-/binding-darwin-arm64-1.51.0.tgz",
|
||||
"integrity": "sha512-3QJbeYaMHn6Bh2XeBXuITSsbnIctyTjvHf5nRjKYrT9pPeErNIpp5VDEeAXC0CZSwSVTsc8WOSDwgrAI24JolQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-darwin-x64": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-darwin-x64/-/binding-darwin-x64-1.51.0.tgz",
|
||||
"integrity": "sha512-NzErhMaTEN1cY0E8C5APy74lw5VwsNfJfVPBMWPVQLqAbO0k4FFLjvHURvkUL+Y18Wu+8Vs1kbqPh2hjXYA4pg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-freebsd-x64": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-freebsd-x64/-/binding-freebsd-x64-1.51.0.tgz",
|
||||
"integrity": "sha512-msAIh3vPAoKoHlOE/oe6Q5C/n9umypv/k81lED82ibrJotn+3YG2Qp1kiR8o/Dg5iOEU97c6tl0utxcyFenpFw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"freebsd"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-arm-gnueabihf": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.51.0.tgz",
|
||||
"integrity": "sha512-CqQPcvqYyMe9ZBot2stjGogEzk1z8gGAngIX7srSzrzexmXixwVxBdFZyxTVM0CjGfDeV+Ru0w25/WNjlMM2Hw==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-arm-musleabihf": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-1.51.0.tgz",
|
||||
"integrity": "sha512-dstrlYQgZMnyOssxSbolGCge/sDbko12N/35RBNuqLpoPbft2aeBidBAb0dvQlyBd9RJ6u8D4o4Eh8Un6iTgyQ==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-arm64-gnu": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.51.0.tgz",
|
||||
"integrity": "sha512-QEjUpXO7d35rP1/raLGGbAsBLLGZIzV3ZbeSjqWlD3oRnxpRIZ6iL4o51XQHkconn3uKssc+1VKdtHJ81BBhDA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-arm64-musl": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.51.0.tgz",
|
||||
"integrity": "sha512-YSJua5irtG4DoMAjUapDTPhkQLHhBIY0G9JqlZS6/SZPzqDkPku/1GdWs0D6h/wyx0Iz31lNCfIaWKBQhzP0wQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-ppc64-gnu": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.51.0.tgz",
|
||||
"integrity": "sha512-7L4Wj2IEUNDETKssB9IDYt16T6WlF+X2jgC/hBq3diGHda9vJLpAgb09+D3quFq7TdkFtI7hwz/jmuQmQFPc1Q==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-riscv64-gnu": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-1.51.0.tgz",
|
||||
"integrity": "sha512-cBUHqtOXy76G41lOB401qpFoKx1xq17qYkhWrLSM7eEjiHM9sOtYqpr6ZdqCnN9s6ZpzudX4EkeHOFH2E9q0vA==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-riscv64-musl": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-1.51.0.tgz",
|
||||
"integrity": "sha512-WKbg8CysgZcHfZX0ixQFBRSBvFZUHa3SBnEjHY2FVYt2nbNJEjzTxA3ZR5wMU0NOCNKIAFUFvAh5/XJKPRJuJg==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-s390x-gnu": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.51.0.tgz",
|
||||
"integrity": "sha512-N1QRUvJTxqXNSu35YOufdjsAVmKVx5bkrggOWAhTWBc3J4qjcBwr1IfyLh/6YCg8sYRSR1GraldS9jUgJL/U4A==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-x64-gnu": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.51.0.tgz",
|
||||
"integrity": "sha512-e0Mz0DizsCoqNIjeOg6OUKe8JKJWZ5zZlwsd05Bmr51Jo3AOL4UJnPvwKumr4BBtBrDZkCmOLhCvDGm95nJM2g==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-linux-x64-musl": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-linux-x64-musl/-/binding-linux-x64-musl-1.51.0.tgz",
|
||||
"integrity": "sha512-wD8HGTWhYBKXvRDvoBVB1y+fEYV01samhWQSy1Zkxq2vpezvMnjaFKRuiP6tBNITLGuffbNDEXOwcAhJ3gI5Ug==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"linux"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-openharmony-arm64": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-openharmony-arm64/-/binding-openharmony-arm64-1.51.0.tgz",
|
||||
"integrity": "sha512-5NSwQ2hDEJ0GPXqikjWtwzgAQCsS7P9aLMNenjjKa+gknN3lTCwwwERsT6lKXSirfU3jLjexA2XQvQALh5h27w==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"openharmony"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-win32-arm64-msvc": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.51.0.tgz",
|
||||
"integrity": "sha512-JEZyah1M0RHMw8d+jjSSJmSmO8sABA1J1RtrHYujGPeCkYg1NeH0TGuClpe2h5QtioRTaF57y/TZfn/2IFV6fA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-win32-ia32-msvc": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-1.51.0.tgz",
|
||||
"integrity": "sha512-q3cEoKH6kwjz/WRyHwSf0nlD2F5Qw536kCXvmlSu+kaShzgrA0ojmh45CA81qL+7udfCaZL2SdKCZlLiGBVFlg==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@oxlint/binding-win32-x64-msvc": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxlint/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.51.0.tgz",
|
||||
"integrity": "sha512-Q14+fOGb9T28nWF/0EUsYqERiRA7cl1oy4TJrGmLaqhm+aO2cV+JttboHI3CbdeMCAyDI1+NoSlrM7Melhp/cw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"win32"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@tybys/wasm-util": {
|
||||
"version": "0.10.1",
|
||||
"resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz",
|
||||
"integrity": "sha512-9tTaPJLSiejZKx+Bmog4uSubteqTvFrVrURwkmHixBo0G4seD0zUxp98E1DzUBJxLQ3NPwXrGKDiVjwx/DpPsg==",
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/oxc-parser": {
|
||||
"version": "0.116.0",
|
||||
"resolved": "https://registry.npmjs.org/oxc-parser/-/oxc-parser-0.116.0.tgz",
|
||||
"integrity": "sha512-ugEo6wwqaqCGcpi7GsLCwSkoD7gIXzvtdaTxE+mbrXFYazU5Q9YdpZdAj9z2b79i/xlv+uW2aAvyzGAlpUzhKQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@oxc-project/types": "^0.116.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/Boshen"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@oxc-parser/binding-android-arm-eabi": "0.116.0",
|
||||
"@oxc-parser/binding-android-arm64": "0.116.0",
|
||||
"@oxc-parser/binding-darwin-arm64": "0.116.0",
|
||||
"@oxc-parser/binding-darwin-x64": "0.116.0",
|
||||
"@oxc-parser/binding-freebsd-x64": "0.116.0",
|
||||
"@oxc-parser/binding-linux-arm-gnueabihf": "0.116.0",
|
||||
"@oxc-parser/binding-linux-arm-musleabihf": "0.116.0",
|
||||
"@oxc-parser/binding-linux-arm64-gnu": "0.116.0",
|
||||
"@oxc-parser/binding-linux-arm64-musl": "0.116.0",
|
||||
"@oxc-parser/binding-linux-ppc64-gnu": "0.116.0",
|
||||
"@oxc-parser/binding-linux-riscv64-gnu": "0.116.0",
|
||||
"@oxc-parser/binding-linux-riscv64-musl": "0.116.0",
|
||||
"@oxc-parser/binding-linux-s390x-gnu": "0.116.0",
|
||||
"@oxc-parser/binding-linux-x64-gnu": "0.116.0",
|
||||
"@oxc-parser/binding-linux-x64-musl": "0.116.0",
|
||||
"@oxc-parser/binding-openharmony-arm64": "0.116.0",
|
||||
"@oxc-parser/binding-wasm32-wasi": "0.116.0",
|
||||
"@oxc-parser/binding-win32-arm64-msvc": "0.116.0",
|
||||
"@oxc-parser/binding-win32-ia32-msvc": "0.116.0",
|
||||
"@oxc-parser/binding-win32-x64-msvc": "0.116.0"
|
||||
}
|
||||
},
|
||||
"node_modules/oxlint": {
|
||||
"version": "1.51.0",
|
||||
"resolved": "https://registry.npmjs.org/oxlint/-/oxlint-1.51.0.tgz",
|
||||
"integrity": "sha512-g6DNPaV9/WI9MoX2XllafxQuxwY1TV++j7hP8fTJByVBuCoVtm3dy9f/2vtH/HU40JztcgWF4G7ua+gkainklQ==",
|
||||
"license": "MIT",
|
||||
"bin": {
|
||||
"oxlint": "bin/oxlint"
|
||||
},
|
||||
"engines": {
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/Boshen"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@oxlint/binding-android-arm-eabi": "1.51.0",
|
||||
"@oxlint/binding-android-arm64": "1.51.0",
|
||||
"@oxlint/binding-darwin-arm64": "1.51.0",
|
||||
"@oxlint/binding-darwin-x64": "1.51.0",
|
||||
"@oxlint/binding-freebsd-x64": "1.51.0",
|
||||
"@oxlint/binding-linux-arm-gnueabihf": "1.51.0",
|
||||
"@oxlint/binding-linux-arm-musleabihf": "1.51.0",
|
||||
"@oxlint/binding-linux-arm64-gnu": "1.51.0",
|
||||
"@oxlint/binding-linux-arm64-musl": "1.51.0",
|
||||
"@oxlint/binding-linux-ppc64-gnu": "1.51.0",
|
||||
"@oxlint/binding-linux-riscv64-gnu": "1.51.0",
|
||||
"@oxlint/binding-linux-riscv64-musl": "1.51.0",
|
||||
"@oxlint/binding-linux-s390x-gnu": "1.51.0",
|
||||
"@oxlint/binding-linux-x64-gnu": "1.51.0",
|
||||
"@oxlint/binding-linux-x64-musl": "1.51.0",
|
||||
"@oxlint/binding-openharmony-arm64": "1.51.0",
|
||||
"@oxlint/binding-win32-arm64-msvc": "1.51.0",
|
||||
"@oxlint/binding-win32-ia32-msvc": "1.51.0",
|
||||
"@oxlint/binding-win32-x64-msvc": "1.51.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"oxlint-tsgolint": ">=0.15.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"oxlint-tsgolint": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/tslib": {
|
||||
"version": "2.8.1",
|
||||
"resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz",
|
||||
"integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==",
|
||||
"license": "0BSD",
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
}
|
||||
10
studio/backend/core/data_recipe/oxc-validator/package.json
Normal file
10
studio/backend/core/data_recipe/oxc-validator/package.json
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"name": "unsloth-oxc-validator-runtime",
|
||||
"private": true,
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"dependencies": {
|
||||
"oxc-parser": "^0.116.0",
|
||||
"oxlint": "^1.51.0"
|
||||
}
|
||||
}
|
||||
576
studio/backend/core/data_recipe/oxc-validator/validate.mjs
Normal file
576
studio/backend/core/data_recipe/oxc-validator/validate.mjs
Normal file
|
|
@ -0,0 +1,576 @@
|
|||
// SPDX-License-Identifier: AGPL-3.0-only
|
||||
// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { basename, dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { parseSync } from "oxc-parser";
|
||||
|
||||
const LANG_TO_EXT = {
|
||||
js: "js",
|
||||
jsx: "jsx",
|
||||
ts: "ts",
|
||||
tsx: "tsx",
|
||||
};
|
||||
|
||||
const VALIDATION_MODES = new Set(["syntax", "lint", "syntax+lint"]);
|
||||
const CODE_SHAPES = new Set(["auto", "module", "snippet"]);
|
||||
const SNIPPET_PREFIX = "(() => {\n";
|
||||
const SNIPPET_SUFFIX = "\n})();\nexport {};\n";
|
||||
const OXLINT_SUPPRESSED_RULES = ["no-unused-vars", "no-new-array"];
|
||||
const TOOL_DIR = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
function mapLang(value) {
|
||||
const normalized = String(value || "").trim().toLowerCase();
|
||||
if (normalized === "javascript" || normalized === "js") {
|
||||
return "js";
|
||||
}
|
||||
if (normalized === "typescript" || normalized === "ts") {
|
||||
return "ts";
|
||||
}
|
||||
if (normalized === "jsx") {
|
||||
return "jsx";
|
||||
}
|
||||
if (normalized === "tsx") {
|
||||
return "tsx";
|
||||
}
|
||||
return "js";
|
||||
}
|
||||
|
||||
function mapMode(value) {
|
||||
const normalized = String(value || "").trim().toLowerCase();
|
||||
if (VALIDATION_MODES.has(normalized)) {
|
||||
return normalized;
|
||||
}
|
||||
return "syntax";
|
||||
}
|
||||
|
||||
function mapCodeShape(value) {
|
||||
const normalized = String(value || "").trim().toLowerCase();
|
||||
if (CODE_SHAPES.has(normalized)) {
|
||||
return normalized;
|
||||
}
|
||||
return "auto";
|
||||
}
|
||||
|
||||
function parseFileIndex(filePath) {
|
||||
if (typeof filePath !== "string") {
|
||||
return null;
|
||||
}
|
||||
const match = basename(filePath).match(/^snippet_(\d+)\./);
|
||||
if (!match) {
|
||||
return null;
|
||||
}
|
||||
const parsed = Number.parseInt(match[1], 10);
|
||||
return Number.isFinite(parsed) ? parsed : null;
|
||||
}
|
||||
|
||||
function toCodeString(code) {
|
||||
return typeof code === "string" ? code : String(code ?? "");
|
||||
}
|
||||
|
||||
function makeValidationEntry({ code, index, lang, codeShape }) {
|
||||
const source = toCodeString(code);
|
||||
if (codeShape === "snippet") {
|
||||
return {
|
||||
index,
|
||||
lang,
|
||||
code: `${SNIPPET_PREFIX}${source}${SNIPPET_SUFFIX}`,
|
||||
offset: SNIPPET_PREFIX.length,
|
||||
};
|
||||
}
|
||||
return {
|
||||
index,
|
||||
lang,
|
||||
code: source,
|
||||
offset: 0,
|
||||
};
|
||||
}
|
||||
|
||||
function shiftOffset(value, offset) {
|
||||
if (!Number.isInteger(value)) {
|
||||
return null;
|
||||
}
|
||||
const shifted = value - offset;
|
||||
return shifted >= 0 ? shifted : null;
|
||||
}
|
||||
|
||||
function remapDiagnosticOffsets(diagnostic, offset) {
|
||||
if (!diagnostic || typeof diagnostic !== "object" || offset <= 0) {
|
||||
return diagnostic;
|
||||
}
|
||||
return {
|
||||
...diagnostic,
|
||||
labels: Array.isArray(diagnostic.labels)
|
||||
? diagnostic.labels.map((label) => ({
|
||||
...label,
|
||||
start: shiftOffset(label.start, offset),
|
||||
end: shiftOffset(label.end, offset),
|
||||
}))
|
||||
: [],
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeParserError(error) {
|
||||
if (typeof error === "string") {
|
||||
return {
|
||||
code: null,
|
||||
message: error.trim() || "Unknown parser error",
|
||||
severity: null,
|
||||
labels: [],
|
||||
codeframe: null,
|
||||
};
|
||||
}
|
||||
if (!error || typeof error !== "object") {
|
||||
return {
|
||||
code: null,
|
||||
message: "Unknown parser error",
|
||||
severity: null,
|
||||
labels: [],
|
||||
codeframe: null,
|
||||
};
|
||||
}
|
||||
const code = typeof error.code === "string" ? error.code : null;
|
||||
const message = String(error.message || error.reason || "").trim() || "Unknown parser error";
|
||||
const severity = typeof error.severity === "string" ? error.severity : null;
|
||||
const labels = Array.isArray(error.labels)
|
||||
? error.labels.map((label) => ({
|
||||
message:
|
||||
label && typeof label === "object" && typeof label.message === "string"
|
||||
? label.message
|
||||
: null,
|
||||
start:
|
||||
label && typeof label === "object" && Number.isInteger(label.start)
|
||||
? label.start
|
||||
: null,
|
||||
end:
|
||||
label && typeof label === "object" && Number.isInteger(label.end)
|
||||
? label.end
|
||||
: null,
|
||||
}))
|
||||
: [];
|
||||
const codeframe = typeof error.codeframe === "string" ? error.codeframe : null;
|
||||
return {
|
||||
code,
|
||||
message,
|
||||
severity,
|
||||
labels,
|
||||
codeframe,
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeLintDiagnostic(diagnostic) {
|
||||
if (!diagnostic || typeof diagnostic !== "object") {
|
||||
return null;
|
||||
}
|
||||
|
||||
const readString = (value) =>
|
||||
typeof value === "string" ? value : null;
|
||||
const readInt = (value) =>
|
||||
Number.isInteger(value) ? value : null;
|
||||
const asObject = (value) =>
|
||||
value && typeof value === "object" ? value : null;
|
||||
|
||||
const message = String(diagnostic.message || "").trim();
|
||||
if (!message) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const severityRaw = String(diagnostic.severity || "").trim().toLowerCase();
|
||||
const severity = severityRaw === "error" ? "error" : "warning";
|
||||
|
||||
const labels = [];
|
||||
if (Array.isArray(diagnostic.labels)) {
|
||||
for (const label of diagnostic.labels) {
|
||||
const labelObj = asObject(label);
|
||||
const span = asObject(labelObj?.span);
|
||||
const start = readInt(span?.offset);
|
||||
const length = readInt(span?.length);
|
||||
labels.push({
|
||||
message: readString(labelObj?.label),
|
||||
start,
|
||||
end: start !== null && length !== null ? start + length : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const code = typeof diagnostic.code === "string" ? diagnostic.code : null;
|
||||
return {
|
||||
code,
|
||||
message: code ? `${code}: ${message}` : message,
|
||||
severity,
|
||||
labels,
|
||||
codeframe: null,
|
||||
};
|
||||
}
|
||||
|
||||
function makeResult({
|
||||
isValid,
|
||||
errorCount,
|
||||
warningCount = 0,
|
||||
message = "",
|
||||
severity = null,
|
||||
code = null,
|
||||
labels = [],
|
||||
codeframe = null,
|
||||
}) {
|
||||
return {
|
||||
is_valid: Boolean(isValid),
|
||||
error_count: Number.isInteger(errorCount) ? errorCount : 0,
|
||||
warning_count: Number.isInteger(warningCount) ? warningCount : 0,
|
||||
error_message: String(message || ""),
|
||||
severity: typeof severity === "string" ? severity : null,
|
||||
code: typeof code === "string" ? code : null,
|
||||
labels: Array.isArray(labels) ? labels : [],
|
||||
codeframe: typeof codeframe === "string" ? codeframe : null,
|
||||
};
|
||||
}
|
||||
|
||||
function syntaxResultFromErrors(errors) {
|
||||
const first = errors[0] ?? null;
|
||||
return makeResult({
|
||||
isValid: errors.length === 0,
|
||||
errorCount: errors.length,
|
||||
warningCount: 0,
|
||||
message: errors.slice(0, 3).map((error) => error.message).join(" | "),
|
||||
severity: first ? first.severity : null,
|
||||
code: first ? first.code : null,
|
||||
labels: first ? first.labels : [],
|
||||
codeframe: first ? first.codeframe : null,
|
||||
});
|
||||
}
|
||||
|
||||
function runSyntaxParse(entry) {
|
||||
const ext = LANG_TO_EXT[entry.lang] ?? "js";
|
||||
const filename = `snippet_${entry.index}.${ext}`;
|
||||
try {
|
||||
const parsed = parseSync(filename, entry.code, {
|
||||
lang: entry.lang,
|
||||
sourceType: "module",
|
||||
showSemanticErrors: true,
|
||||
});
|
||||
const errors = Array.isArray(parsed?.errors)
|
||||
? parsed.errors
|
||||
.map(normalizeParserError)
|
||||
.filter(Boolean)
|
||||
.map((error) => remapDiagnosticOffsets(error, entry.offset))
|
||||
: [];
|
||||
return errors;
|
||||
} catch (error) {
|
||||
return [
|
||||
remapDiagnosticOffsets(
|
||||
normalizeParserError(error),
|
||||
entry.offset,
|
||||
),
|
||||
];
|
||||
}
|
||||
}
|
||||
|
||||
function pickPreferredErrorList(firstErrors, secondErrors) {
|
||||
if (secondErrors.length < firstErrors.length) {
|
||||
return secondErrors;
|
||||
}
|
||||
return firstErrors;
|
||||
}
|
||||
|
||||
function validateSyntaxOne({ code, lang, index, codeShape }) {
|
||||
if (codeShape !== "auto") {
|
||||
const lintEntry = makeValidationEntry({
|
||||
code,
|
||||
index,
|
||||
lang,
|
||||
codeShape,
|
||||
});
|
||||
const errors = runSyntaxParse(lintEntry);
|
||||
return {
|
||||
result: syntaxResultFromErrors(errors),
|
||||
lintEntry,
|
||||
};
|
||||
}
|
||||
|
||||
const moduleEntry = makeValidationEntry({
|
||||
code,
|
||||
index,
|
||||
lang,
|
||||
codeShape: "module",
|
||||
});
|
||||
const moduleErrors = runSyntaxParse(moduleEntry);
|
||||
if (moduleErrors.length === 0) {
|
||||
return {
|
||||
result: syntaxResultFromErrors(moduleErrors),
|
||||
lintEntry: moduleEntry,
|
||||
};
|
||||
}
|
||||
|
||||
const snippetEntry = makeValidationEntry({
|
||||
code,
|
||||
index,
|
||||
lang,
|
||||
codeShape: "snippet",
|
||||
});
|
||||
const snippetErrors = runSyntaxParse(snippetEntry);
|
||||
if (snippetErrors.length === 0) {
|
||||
return {
|
||||
result: syntaxResultFromErrors(snippetErrors),
|
||||
lintEntry: snippetEntry,
|
||||
};
|
||||
}
|
||||
|
||||
const chosenErrors = pickPreferredErrorList(moduleErrors, snippetErrors);
|
||||
const lintEntry = chosenErrors === snippetErrors ? snippetEntry : moduleEntry;
|
||||
return {
|
||||
result: syntaxResultFromErrors(chosenErrors),
|
||||
lintEntry,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveLintEntry({ code, lang, index, codeShape }) {
|
||||
if (codeShape !== "auto") {
|
||||
return makeValidationEntry({
|
||||
code,
|
||||
index,
|
||||
lang,
|
||||
codeShape,
|
||||
});
|
||||
}
|
||||
|
||||
const moduleEntry = makeValidationEntry({
|
||||
code,
|
||||
index,
|
||||
lang,
|
||||
codeShape: "module",
|
||||
});
|
||||
if (runSyntaxParse(moduleEntry).length === 0) {
|
||||
return moduleEntry;
|
||||
}
|
||||
|
||||
const snippetEntry = makeValidationEntry({
|
||||
code,
|
||||
index,
|
||||
lang,
|
||||
codeShape: "snippet",
|
||||
});
|
||||
if (runSyntaxParse(snippetEntry).length === 0) {
|
||||
return snippetEntry;
|
||||
}
|
||||
|
||||
return moduleEntry;
|
||||
}
|
||||
|
||||
function fallbackLintResults(entries, message) {
|
||||
return new Map(
|
||||
entries.map((entry) => [
|
||||
entry.index,
|
||||
makeResult({
|
||||
isValid: false,
|
||||
errorCount: 1,
|
||||
warningCount: 0,
|
||||
message,
|
||||
severity: "error",
|
||||
}),
|
||||
]),
|
||||
);
|
||||
}
|
||||
|
||||
function runLintBatch(entries) {
|
||||
if (entries.length === 0) {
|
||||
return new Map();
|
||||
}
|
||||
|
||||
const entryByIndex = new Map(entries.map((entry) => [entry.index, entry]));
|
||||
const tempDir = mkdtempSync(join(tmpdir(), "oxlint-"));
|
||||
try {
|
||||
for (const entry of entries) {
|
||||
const ext = LANG_TO_EXT[entry.lang] ?? "js";
|
||||
const filePath = join(tempDir, `snippet_${entry.index}.${ext}`);
|
||||
writeFileSync(filePath, entry.code, "utf8");
|
||||
}
|
||||
|
||||
const oxlintBin = join(TOOL_DIR, "node_modules", ".bin", "oxlint");
|
||||
const oxlintArgs = [
|
||||
...OXLINT_SUPPRESSED_RULES.flatMap((rule) => ["-A", rule]),
|
||||
"--format",
|
||||
"json",
|
||||
tempDir,
|
||||
];
|
||||
const exec = spawnSync(oxlintBin, oxlintArgs, {
|
||||
encoding: "utf8",
|
||||
cwd: TOOL_DIR,
|
||||
});
|
||||
if (exec.error) {
|
||||
return fallbackLintResults(
|
||||
entries,
|
||||
`oxlint execution failed: ${exec.error.message}`,
|
||||
);
|
||||
}
|
||||
const stdout = String(exec.stdout || "").trim();
|
||||
if (!stdout) {
|
||||
const stderr = String(exec.stderr || "").trim();
|
||||
return fallbackLintResults(
|
||||
entries,
|
||||
stderr || "oxlint returned empty output",
|
||||
);
|
||||
}
|
||||
|
||||
let parsed;
|
||||
try {
|
||||
parsed = JSON.parse(stdout);
|
||||
} catch {
|
||||
return fallbackLintResults(entries, "oxlint JSON parse failed");
|
||||
}
|
||||
|
||||
const rawDiagnostics = Array.isArray(parsed?.diagnostics)
|
||||
? parsed.diagnostics
|
||||
: [];
|
||||
const byIndex = new Map();
|
||||
|
||||
for (const diag of rawDiagnostics) {
|
||||
const filenameRaw =
|
||||
typeof diag?.filename === "string" ? diag.filename : "";
|
||||
const filename = filenameRaw.startsWith("file://")
|
||||
? filenameRaw.replace("file://", "")
|
||||
: filenameRaw;
|
||||
const index = parseFileIndex(filename);
|
||||
if (index === null) {
|
||||
continue;
|
||||
}
|
||||
const normalized = normalizeLintDiagnostic(diag);
|
||||
if (!normalized) {
|
||||
continue;
|
||||
}
|
||||
const entry = entryByIndex.get(index);
|
||||
const remapped = remapDiagnosticOffsets(normalized, entry?.offset ?? 0);
|
||||
const list = byIndex.get(index) ?? [];
|
||||
list.push(remapped);
|
||||
byIndex.set(index, list);
|
||||
}
|
||||
|
||||
const results = new Map();
|
||||
for (const entry of entries) {
|
||||
const diagnostics = byIndex.get(entry.index) ?? [];
|
||||
const errorDiagnostics = diagnostics.filter(
|
||||
(diag) => diag.severity === "error",
|
||||
);
|
||||
const warningDiagnostics = diagnostics.filter(
|
||||
(diag) => diag.severity !== "error",
|
||||
);
|
||||
const top = errorDiagnostics[0] ?? warningDiagnostics[0] ?? null;
|
||||
const messageSource =
|
||||
errorDiagnostics.length > 0 ? errorDiagnostics : warningDiagnostics;
|
||||
results.set(
|
||||
entry.index,
|
||||
makeResult({
|
||||
isValid: errorDiagnostics.length === 0,
|
||||
errorCount: errorDiagnostics.length,
|
||||
warningCount: warningDiagnostics.length,
|
||||
message: messageSource
|
||||
.slice(0, 3)
|
||||
.map((diag) => diag.message)
|
||||
.join(" | "),
|
||||
severity: top ? top.severity : null,
|
||||
code: top ? top.code : null,
|
||||
labels: top ? top.labels : [],
|
||||
codeframe: top ? top.codeframe : null,
|
||||
}),
|
||||
);
|
||||
}
|
||||
return results;
|
||||
} catch (error) {
|
||||
return fallbackLintResults(entries, `oxlint execution failed: ${error}`);
|
||||
} finally {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function readStdin() {
|
||||
return new Promise((resolve, reject) => {
|
||||
let data = "";
|
||||
process.stdin.setEncoding("utf8");
|
||||
process.stdin.on("data", (chunk) => {
|
||||
data += chunk;
|
||||
});
|
||||
process.stdin.on("end", () => resolve(data));
|
||||
process.stdin.on("error", (error) => reject(error));
|
||||
});
|
||||
}
|
||||
|
||||
function runValidation({ codes, lang, mode, codeShape }) {
|
||||
if (mode === "syntax") {
|
||||
return codes.map((code, index) =>
|
||||
validateSyntaxOne({ code, lang, index, codeShape }).result,
|
||||
);
|
||||
}
|
||||
|
||||
if (mode === "lint") {
|
||||
const entries = codes.map((code, index) =>
|
||||
resolveLintEntry({ code, lang, index, codeShape }),
|
||||
);
|
||||
const lintMap = runLintBatch(entries);
|
||||
return entries.map(
|
||||
(entry) =>
|
||||
lintMap.get(entry.index) ??
|
||||
makeResult({
|
||||
isValid: true,
|
||||
errorCount: 0,
|
||||
warningCount: 0,
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
const syntaxRuns = codes.map((code, index) =>
|
||||
validateSyntaxOne({ code, lang, index, codeShape }),
|
||||
);
|
||||
const lintTargets = syntaxRuns
|
||||
.filter((run) => run.result.is_valid === true)
|
||||
.map((run) => run.lintEntry);
|
||||
const lintMap = runLintBatch(lintTargets);
|
||||
|
||||
return syntaxRuns.map((run) => {
|
||||
if (run.result.is_valid !== true) {
|
||||
return run.result;
|
||||
}
|
||||
return (
|
||||
lintMap.get(run.lintEntry.index) ??
|
||||
makeResult({
|
||||
isValid: true,
|
||||
errorCount: 0,
|
||||
warningCount: 0,
|
||||
})
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const raw = await readStdin();
|
||||
let payload;
|
||||
try {
|
||||
payload = JSON.parse(raw || "{}");
|
||||
} catch {
|
||||
process.stdout.write(
|
||||
JSON.stringify([
|
||||
makeResult({
|
||||
isValid: false,
|
||||
errorCount: 1,
|
||||
warningCount: 0,
|
||||
message: "Invalid JSON payload",
|
||||
severity: "error",
|
||||
}),
|
||||
]),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const lang = mapLang(payload?.lang);
|
||||
const mode = mapMode(payload?.mode);
|
||||
const codeShape = mapCodeShape(payload?.code_shape);
|
||||
const codes = Array.isArray(payload?.codes) ? payload.codes : [];
|
||||
const out = runValidation({ codes, lang, mode, codeShape });
|
||||
process.stdout.write(JSON.stringify(out));
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
process.stderr.write(String(error?.stack || error));
|
||||
process.exit(1);
|
||||
});
|
||||
283
studio/backend/core/data_recipe/service.py
Normal file
283
studio/backend/core/data_recipe/service.py
Normal file
|
|
@ -0,0 +1,283 @@
|
|||
# SPDX-License-Identifier: AGPL-3.0-only
|
||||
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import io
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .jsonable import to_jsonable
|
||||
from .local_callable_validators import (
|
||||
register_oxc_local_callable_validators,
|
||||
split_oxc_local_callable_validators,
|
||||
)
|
||||
|
||||
_IMAGE_CONTEXT_PATCHED = False
|
||||
|
||||
|
||||
def _encode_bytes_to_base64(value: bytes | bytearray) -> str:
|
||||
return base64.b64encode(bytes(value)).decode("utf-8")
|
||||
|
||||
|
||||
def _load_image_file_to_base64(
|
||||
path_value: str, *, base_path: str | None = None
|
||||
) -> str | None:
|
||||
try:
|
||||
path = Path(path_value)
|
||||
candidates: list[Path] = []
|
||||
if path.is_absolute():
|
||||
candidates.append(path)
|
||||
else:
|
||||
if base_path:
|
||||
candidates.append(Path(base_path) / path)
|
||||
candidates.append(Path.cwd() / path)
|
||||
|
||||
for candidate in candidates:
|
||||
if not candidate.exists() or not candidate.is_file():
|
||||
continue
|
||||
with candidate.open("rb") as f:
|
||||
return _encode_bytes_to_base64(f.read())
|
||||
except (OSError, TypeError, ValueError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _pil_image_to_base64(value: Any) -> str | None:
|
||||
try:
|
||||
from PIL.Image import Image as PILImage # type: ignore
|
||||
except ImportError:
|
||||
return None
|
||||
if not isinstance(value, PILImage):
|
||||
return None
|
||||
buffer = io.BytesIO()
|
||||
image_format = str(getattr(value, "format", "") or "").upper()
|
||||
if image_format not in {"PNG", "JPEG", "JPG", "WEBP", "GIF"}:
|
||||
image_format = "PNG"
|
||||
value.save(buffer, format = image_format)
|
||||
return _encode_bytes_to_base64(buffer.getvalue())
|
||||
|
||||
|
||||
def _normalize_image_context_value(value: Any, *, base_path: str | None = None) -> Any:
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
|
||||
if isinstance(value, (bytes, bytearray)):
|
||||
return _encode_bytes_to_base64(value)
|
||||
|
||||
pil_base64 = _pil_image_to_base64(value)
|
||||
if pil_base64 is not None:
|
||||
return pil_base64
|
||||
|
||||
if isinstance(value, dict):
|
||||
url = value.get("url")
|
||||
if isinstance(url, str):
|
||||
return url
|
||||
|
||||
image_url = value.get("image_url")
|
||||
if isinstance(image_url, str):
|
||||
return image_url
|
||||
if isinstance(image_url, dict):
|
||||
nested_url = image_url.get("url")
|
||||
if isinstance(nested_url, str):
|
||||
return nested_url
|
||||
|
||||
inline_data = value.get("data")
|
||||
if isinstance(inline_data, str):
|
||||
return inline_data
|
||||
|
||||
raw_bytes = value.get("bytes")
|
||||
if isinstance(raw_bytes, (bytes, bytearray)):
|
||||
return _encode_bytes_to_base64(raw_bytes)
|
||||
if isinstance(raw_bytes, str) and raw_bytes.strip():
|
||||
return raw_bytes
|
||||
|
||||
path_value = value.get("path")
|
||||
if isinstance(path_value, str) and path_value.strip():
|
||||
if as_base64 := _load_image_file_to_base64(path_value, base_path = base_path):
|
||||
return as_base64
|
||||
return path_value
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def _apply_data_designer_image_context_patch() -> None:
|
||||
global _IMAGE_CONTEXT_PATCHED
|
||||
if _IMAGE_CONTEXT_PATCHED:
|
||||
return
|
||||
|
||||
try:
|
||||
from data_designer.config.models import ImageContext
|
||||
except ImportError:
|
||||
return
|
||||
|
||||
if getattr(ImageContext, "_unsloth_image_context_patch_applied", False):
|
||||
_IMAGE_CONTEXT_PATCHED = True
|
||||
return
|
||||
|
||||
original_auto_resolve = ImageContext._auto_resolve_context_value
|
||||
|
||||
def _patched_auto_resolve(
|
||||
self: Any, context_value: Any, base_path: str | None
|
||||
) -> Any:
|
||||
normalized = _normalize_image_context_value(context_value, base_path = base_path)
|
||||
return original_auto_resolve(self, normalized, base_path)
|
||||
|
||||
ImageContext._auto_resolve_context_value = _patched_auto_resolve
|
||||
setattr(ImageContext, "_unsloth_image_context_patch_applied", True)
|
||||
_IMAGE_CONTEXT_PATCHED = True
|
||||
|
||||
|
||||
def build_model_providers(recipe: dict[str, Any]):
|
||||
from data_designer.config.default_model_settings import get_default_providers
|
||||
from data_designer.config.models import ModelProvider
|
||||
|
||||
providers: list[ModelProvider] = []
|
||||
for provider in recipe.get("model_providers", []):
|
||||
api_key = provider.get("api_key")
|
||||
api_key_env = provider.get("api_key_env")
|
||||
if not api_key and api_key_env:
|
||||
api_key = os.getenv(api_key_env)
|
||||
providers.append(
|
||||
ModelProvider(
|
||||
name = provider["name"],
|
||||
endpoint = provider["endpoint"],
|
||||
provider_type = provider.get("provider_type", "openai"),
|
||||
api_key = api_key,
|
||||
extra_headers = provider.get("extra_headers"),
|
||||
extra_body = provider.get("extra_body"),
|
||||
)
|
||||
)
|
||||
|
||||
# DataDesigner currently expects at least one provider even if they only use static samplers,
|
||||
# but it's fine it gives a warning only.
|
||||
return providers or get_default_providers()
|
||||
|
||||
|
||||
def build_mcp_providers(
|
||||
recipe: dict[str, Any],
|
||||
) -> list:
|
||||
from data_designer.config.mcp import LocalStdioMCPProvider, MCPProvider
|
||||
|
||||
providers: list[MCPProvider | LocalStdioMCPProvider] = []
|
||||
for provider in recipe.get("mcp_providers", []):
|
||||
if not isinstance(provider, dict):
|
||||
continue
|
||||
provider_type = provider.get("provider_type")
|
||||
if provider_type == "stdio":
|
||||
env = provider.get("env")
|
||||
if not isinstance(env, dict):
|
||||
env = {}
|
||||
args = provider.get("args")
|
||||
if not isinstance(args, list):
|
||||
args = []
|
||||
providers.append(
|
||||
LocalStdioMCPProvider(
|
||||
name = str(provider.get("name", "")),
|
||||
command = str(provider.get("command", "")),
|
||||
args = [str(value) for value in args],
|
||||
env = {str(key): str(value) for key, value in env.items()},
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
if provider_type in {"sse", "streamable_http"}:
|
||||
api_key = provider.get("api_key")
|
||||
api_key_env = provider.get("api_key_env")
|
||||
if not api_key and api_key_env:
|
||||
api_key = os.getenv(str(api_key_env))
|
||||
providers.append(
|
||||
MCPProvider(
|
||||
name = str(provider.get("name", "")),
|
||||
endpoint = str(provider.get("endpoint", "")),
|
||||
provider_type = str(provider_type),
|
||||
api_key = str(api_key) if api_key else None,
|
||||
)
|
||||
)
|
||||
return providers
|
||||
|
||||
|
||||
def build_config_builder(recipe: dict[str, Any]):
|
||||
_apply_data_designer_image_context_patch()
|
||||
from data_designer.config import DataDesignerConfigBuilder
|
||||
from data_designer.config.processors import ProcessorType
|
||||
|
||||
recipe_core = {
|
||||
key: value
|
||||
for key, value in recipe.items()
|
||||
if key not in {"model_providers", "mcp_providers"}
|
||||
}
|
||||
recipe_core, oxc_local_callable_specs = split_oxc_local_callable_validators(
|
||||
recipe_core
|
||||
)
|
||||
builder = DataDesignerConfigBuilder.from_config({"data_designer": recipe_core})
|
||||
register_oxc_local_callable_validators(
|
||||
builder = builder,
|
||||
specs = oxc_local_callable_specs,
|
||||
)
|
||||
|
||||
# DataDesignerConfigBuilder.from_config currently skips processors.
|
||||
# Re-attach explicitly so drop_columns/schema_transform survive API payload.
|
||||
for processor in recipe_core.get("processors") or []:
|
||||
if not isinstance(processor, dict):
|
||||
continue
|
||||
processor_type_raw = processor.get("processor_type")
|
||||
if not isinstance(processor_type_raw, str):
|
||||
continue
|
||||
kwargs = {k: v for k, v in processor.items() if k != "processor_type"}
|
||||
builder.add_processor(
|
||||
processor_type = ProcessorType(processor_type_raw),
|
||||
**kwargs,
|
||||
)
|
||||
|
||||
return builder
|
||||
|
||||
|
||||
def create_data_designer(
|
||||
recipe: dict[str, Any],
|
||||
*,
|
||||
artifact_path: str | None = None,
|
||||
):
|
||||
_apply_data_designer_image_context_patch()
|
||||
from data_designer.interface.data_designer import DataDesigner
|
||||
|
||||
return DataDesigner(
|
||||
artifact_path = artifact_path,
|
||||
model_providers = build_model_providers(recipe),
|
||||
mcp_providers = build_mcp_providers(recipe),
|
||||
)
|
||||
|
||||
|
||||
def validate_recipe(recipe: dict[str, Any]) -> None:
|
||||
builder = build_config_builder(recipe)
|
||||
designer = create_data_designer(recipe)
|
||||
designer.validate(builder)
|
||||
|
||||
|
||||
def preview_recipe(
|
||||
recipe: dict[str, Any],
|
||||
num_records: int,
|
||||
) -> tuple[list[dict[str, Any]], dict[str, Any] | None, dict[str, Any] | None]:
|
||||
builder = build_config_builder(recipe)
|
||||
designer = create_data_designer(recipe)
|
||||
results = designer.preview(builder, num_records = num_records)
|
||||
|
||||
dataset: list[dict[str, Any]] = []
|
||||
if results.dataset is not None:
|
||||
raw_rows = results.dataset.to_dict(orient = "records")
|
||||
dataset = [to_jsonable(row) for row in raw_rows]
|
||||
|
||||
artifacts = (
|
||||
None
|
||||
if results.processor_artifacts is None
|
||||
else to_jsonable(results.processor_artifacts)
|
||||
)
|
||||
analysis = (
|
||||
None
|
||||
if results.analysis is None
|
||||
else to_jsonable(results.analysis.model_dump(mode = "json"))
|
||||
)
|
||||
|
||||
return dataset, artifacts, analysis
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue